-- Logs begin at Sun 2022-07-24 00:10:59 UTC, end at Sun 2022-07-24 01:21:03 UTC. -- Jul 24 00:46:16.342954 dv-cinder-master-3-iscsi systemd[1]: Starting Devstack devstack@keystone.service... Jul 24 00:46:16.349915 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: [uWSGI] getting INI configuration from /etc/keystone/keystone-uwsgi-public.ini Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: *** Starting uWSGI 2.0.18-debian (64bit) on [Sun Jul 24 00:46:16 2022] *** Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: compiled with version: 10.0.1 20200405 (experimental) [master revision 0be9efad938:fcb98e4978a:705510a708d3642c9c962beb663c476167e4e8a4] on 11 April 2020 11:15:55 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: os: Linux-5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: nodename: dv-cinder-master-3-iscsi Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: machine: x86_64 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: clock source: unix Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: pcre jit disabled Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: detected number of CPU cores: 4 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: current working directory: / Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: detected binary path: /usr/bin/uwsgi-core Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: your processes number limit is 47680 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: your memory page size is 4096 bytes Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: detected max file descriptor number: 2048 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: lock engine: pthread robust mutexes Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: thunder lock: enabled Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: uwsgi socket 0 bound to UNIX address /var/run/uwsgi/keystone-wsgi-public.socket fd 4 Jul 24 00:46:16.354802 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: Python version: 3.8.10 (default, Jun 22 2022, 20:18:18) [GCC 9.4.0] Jul 24 00:46:16.397091 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: Python main interpreter initialized at 0x55628393a9c0 Jul 24 00:46:16.397091 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: python threads support enabled Jul 24 00:46:16.397091 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: your server socket listen backlog is limited to 100 connections Jul 24 00:46:16.397091 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: your mercy for graceful operations on workers is 90 seconds Jul 24 00:46:16.397483 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: mapped 403077 bytes (393 KB) for 2 cores Jul 24 00:46:16.397566 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: *** Operational MODE: preforking *** Jul 24 00:46:16.398012 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: *** uWSGI is running in multiple interpreter mode *** Jul 24 00:46:16.398012 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: spawned uWSGI master process (pid: 80223) Jul 24 00:46:16.398214 dv-cinder-master-3-iscsi systemd[1]: Started Devstack devstack@keystone.service. Jul 24 00:46:16.399083 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: spawned uWSGI worker 1 (pid: 80232, cores: 1) Jul 24 00:46:16.399083 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: spawned uWSGI worker 2 (pid: 80233, cores: 1) Jul 24 00:46:16.399255 dv-cinder-master-3-iscsi devstack@keystone.service[80223]: running "unix_signal:15 gracefully_kill_them_all" (master-start)... Jul 24 00:46:21.850743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 24 00:46:21.850743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: from cryptography.utils import int_from_bytes Jul 24 00:46:21.851530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: /usr/lib/python3/dist-packages/secretstorage/dhcrypto.py:15: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 24 00:46:21.851530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: from cryptography.utils import int_from_bytes Jul 24 00:46:21.851941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 24 00:46:21.851941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: from cryptography.utils import int_from_bytes Jul 24 00:46:21.852048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: /usr/lib/python3/dist-packages/secretstorage/util.py:19: CryptographyDeprecationWarning: int_from_bytes is deprecated, use int.from_bytes instead Jul 24 00:46:21.852048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: from cryptography.utils import int_from_bytes Jul 24 00:46:21.900363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 24 00:46:21.900547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 24 00:46:21.900746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] command line args: [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 24 00:46:21.901012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 24 00:46:21.901181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2589}} Jul 24 00:46:21.901270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ================================================================================ {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 24 00:46:21.901450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] Configuration options gathered from: {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2590}} Jul 24 00:46:21.901542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] admin_token = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.901667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] command line args: [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2591}} Jul 24 00:46:21.901753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] config_dir = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.901876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.901959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] config files: ['/etc/keystone/keystone.conf'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2592}} Jul 24 00:46:21.902075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] config_source = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ================================================================================ {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2594}} Jul 24 00:46:21.902284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] debug = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] admin_token = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] config_dir = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.902928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] config_file = ['/etc/keystone/keystone.conf'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] config_source = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] debug = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] default_log_levels = ['amqp=WARN', 'amqplib=WARN', 'boto=WARN', 'qpid=WARN', 'sqlalchemy=WARN', 'suds=INFO', 'oslo.messaging=INFO', 'oslo_messaging=INFO', 'iso8601=WARN', 'requests.packages.urllib3.connectionpool=WARN', 'urllib3.connectionpool=WARN', 'websocket=WARN', 'requests.packages.urllib3.util.retry=WARN', 'urllib3.util.retry=WARN', 'keystonemiddleware=WARN', 'routes.middleware=WARN', 'stevedore=WARN', 'taskflow=WARN', 'keystoneauth=WARN', 'oslo.cache=INFO', 'oslo_policy=INFO', 'dogpile.core.dogpile=INFO', 'dogpile=INFO', 'routes=INFO'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_config_append = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.903900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] default_publisher_id = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] insecure_debug = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_dir = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] instance_format = [instance: %(uuid)s] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_file = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] instance_uuid_format = [instance: %(uuid)s] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.904924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_config_append = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_date_format = %Y-%m-%d %H:%M:%S {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_dir = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_file = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_rotate_interval = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.905965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_rotate_interval_type = days {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] log_rotation_type = none {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] logging_context_format_string = %(color)s%(levelname)s %(name)s [%(global_request_id)s %(request_id)s %(project_name)s %(user_name)s%(color)s] %(instance)s%(color)s%(message)s {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] logging_debug_format_suffix = {{(pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d}} {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] logging_default_format_string = %(color)s%(levelname)s %(name)s [-%(color)s] %(instance)s%(color)s%(message)s {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.906986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] logging_exception_prefix = ERROR %(name)s %(instance)s {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] logging_user_identity_format = %(user)s %(project)s %(domain)s %(system_scope)s %(user_domain)s %(project_domain)s {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] max_logfile_count = 30 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] max_logfile_size_mb = 200 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.907913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] max_param_size = 64 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] max_project_tree_depth = 5 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.18/identity {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] max_token_size = 16384 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] publish_errors = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] notification_format = cadf {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.908968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] notification_opt_out = ['identity.authenticate.success', 'identity.authenticate.pending', 'identity.authenticate.failed'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] public_endpoint = http://10.222.0.18/identity {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] publish_errors = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] pydev_debug_host = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.909919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] standard_threads = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] pydev_debug_port = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] rate_limit_burst = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] rate_limit_except_level = CRITICAL {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] rate_limit_interval = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] use_journal = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] use_json = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.910935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] standard_threads = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] use_stderr = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] strict_password_check = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] use_syslog = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] syslog_log_facility = LOG_USER {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] use_eventlog = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] use_journal = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.911871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] use_json = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.911998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.912081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] use_stderr = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.912208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] use_syslog = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.912290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.912438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] watch_log_file = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2602}} Jul 24 00:46:21.912522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.912692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_concurrency.disable_process_locking = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.912770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.912932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_concurrency.lock_path = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.connection_string = messaging:// {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.enabled = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.es_doc_type = notification {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.913926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.es_scroll_size = 10000 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.es_scroll_time = 2m {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.filter_error_trace = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.hmac_keys = SECRET_KEY {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.914933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.sentinel_service_name = mymaster {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.socket_timeout = 0.1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] profiler.trace_sqlalchemy = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_buffer_size = 1000 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_enabled = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.915989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_process_name = {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_socket_file = /var/tmp/metrics_collector.sock {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_messaging_metrics.metrics_thread_stop_timeout = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.916905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_middleware.http_basic_auth_user_file = /etc/htpasswd {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.917504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.917699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] audit.namespace = openstack {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.917888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.917982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.enforce_new_defaults = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.enforce_scope = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.policy_default_rule = default {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.policy_dirs = ['policy.d'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.918913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.policy_file = policy.yaml {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.remote_content_type = application/x-www-form-urlencoded {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.remote_ssl_ca_crt_file = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_crt_file = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.remote_ssl_client_key_file = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.919951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oslo_policy.remote_ssl_verify_server_crt = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.admin_password = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.admin_tenant_name = admin {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.920910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.admin_token = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.admin_user = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_admin_prefix = {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_host = 127.0.0.1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_port = 35357 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.921892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_protocol = https {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_section = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_type = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_uri = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.922894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.auth_version = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923092 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.cache = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.cafile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.certfile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.923939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.delay_auth_decision = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.enforce_token_bind = permissive {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.http_connect_timeout = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.http_request_max_retries = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.identity_uri = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.924974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.include_service_catalog = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.insecure = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.interface = internal {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.keyfile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.925907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_conn_get_timeout = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_dead_retry = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_maxsize = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.926874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_socket_timeout = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_pool_unused_timeout = 60 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_secret_key = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_security_strategy = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcache_use_advanced_pool = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.927884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.memcached_servers = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.region_name = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles = ['service'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.service_token_roles_required = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.928901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.service_type = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.token_cache_time = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] keystone_authtoken.www_authenticate_uri = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.backend = dogpile.cache.memcached {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.929865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.backend_argument = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.config_prefix = cache.oslo {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.dead_timeout = 60.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.debug_cache_backend = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.enable_retry_client = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.930976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.enable_socket_keepalive = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.enabled = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.expiration_time = 600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.hashclient_retry_attempts = 2 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.931996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.hashclient_retry_delay = 1.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.932146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.932226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_dead_retry = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.932377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.932455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_pool_connection_get_timeout = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.932655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_pool_flush_on_reconnect = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_pool_maxsize = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_pool_unused_timeout = 60 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.933991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_servers = ['localhost:11211'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.memcache_socket_timeout = 1.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.proxies = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.934719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.retry_attempts = 2 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.retry_delay = 0.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.socket_keepalive_count = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.socket_keepalive_idle = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.935948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.socket_keepalive_interval = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.tls_allowed_ciphers = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.tls_cafile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.tls_certfile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.936951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.tls_enabled = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] cache.tls_keyfile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] application_credential.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] application_credential.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] application_credential.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.937888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] application_credential.user_limit = -1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] assignment.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.external = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] assignment.prohibited_implied_role = ['admin'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.938851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.application_credential = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.external = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.mapped = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.password = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.methods = ['external', 'password', 'token', 'oauth1', 'mapped', 'application_credential'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] auth.token = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.oauth1 = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.939960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.password = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] auth.token = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] catalog.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] catalog.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.940995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] catalog.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] catalog.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] catalog.template_file = default_catalog.templates {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.auth_ttl = 15 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.941929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.key_repository = /etc/keystone/credential-keys/ {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.942901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.provider = fernet {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] credential.user_limit = -1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] domain_config.cache_time = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] domain_config.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] domain_config.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.943875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] endpoint_filter.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] endpoint_filter.return_all_endpoints_if_no_filter = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] endpoint_policy.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.944885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] eventlet_server.admin_bind_host = 0.0.0.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] eventlet_server.admin_port = 35357 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] eventlet_server.public_bind_host = 0.0.0.0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] eventlet_server.public_port = 5000 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.945866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.assertion_prefix = {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.default_authorization_ttl = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.federated_domain_name = Federated {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.946994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.remote_id_attribute = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.sso_callback_template = /etc/keystone/sso_callback_template.html {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] federation.trusted_dashboard = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] fernet_receipts.key_repository = /etc/keystone/fernet-keys/ {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.947950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] fernet_receipts.max_active_keys = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] fernet_tokens.key_repository = /etc/keystone/fernet-keys/ {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] fernet_tokens.max_active_keys = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.cache_time = 600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.948971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.default_domain_id = default {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.domain_config_dir = /etc/keystone/domains {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.domain_configurations_from_database = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.949974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.domain_specific_drivers_enabled = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.950960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.max_password_length = 4096 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.password_hash_algorithm = bcrypt {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.password_hash_rounds = 4 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.salt_bytesize = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.951975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.scrypt_block_size = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity.scrypt_parallelism = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity_mapping.backward_compatible_ids = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity_mapping.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.952948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] identity_mapping.generator = sha256 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] jwt_tokens.jws_private_key_repository = /etc/keystone/jws-keys/private {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] jwt_tokens.jws_public_key_repository = /etc/keystone/jws-keys/public {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.alias_dereferencing = default {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.953912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.auth_pool_connection_lifetime = 60 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.auth_pool_size = 100 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.chase_referrals = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.connection_timeout = -1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.debug_level = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.954855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_ad_nesting = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_additional_attribute_mapping = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_attribute_ignore = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_desc_attribute = description {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.955968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_filter = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_id_attribute = cn {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_member_attribute = member {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_members_are_ids = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.956987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_name_attribute = ou {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_objectclass = groupOfNames {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.group_tree_dn = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.page_size = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.password = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.957917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.pool_connection_lifetime = 600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.pool_connection_timeout = -1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.pool_retry_delay = 0.1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.pool_retry_max = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.958907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.pool_size = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.query_scope = one {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.suffix = cn=example,cn=com {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.tls_cacertdir = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.959972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.tls_cacertfile = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.tls_req_cert = demand {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.url = ldap://localhost {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.use_auth_pool = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.960971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.use_pool = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.use_tls = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_additional_attribute_mapping = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.961929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_attribute_ignore = ['default_project_id'] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_default_project_id_attribute = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_description_attribute = description {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_attribute = enabled {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.962871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_default = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_emulation = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_emulation_dn = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_emulation_use_group_config = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_invert = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.963989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_enabled_mask = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_filter = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_id_attribute = cn {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.964986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_mail_attribute = mail {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_name_attribute = sn {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_objectclass = inetOrgPerson {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_pass_attribute = userPassword {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.965983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ldap.user_tree_dn = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] memcache.dead_retry = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] memcache.pool_connection_get_timeout = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.966903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] memcache.pool_maxsize = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] memcache.pool_unused_timeout = 60 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] memcache.socket_timeout = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oauth1.access_token_duration = 86400 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.967952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oauth1.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] oauth1.request_token_duration = 28800 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] policy.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] policy.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.968959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] receipt.cache_on_issue = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] receipt.cache_time = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] receipt.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] receipt.expiration = 300 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.969824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] receipt.provider = fernet {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.admin_project_domain_name = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.admin_project_name = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.970944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.domain_name_url_safe = off {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] resource.project_name_url_safe = off {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] revoke.cache_time = 3600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.971868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] role.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] revoke.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] role.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] revoke.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] revoke.expiration_buffer = 1800 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] role.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] role.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.972963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] role.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] role.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.assertion_expiration_time = 3600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.certfile = /etc/keystone/ssl/certs/signing_cert.pem {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.973927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_company = Example, Inc. {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_email = support@example.com {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_name = SAML Identity Provider Support {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_surname = Support {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.974907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_telephone = +1 800 555 0100 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_contact_type = other {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_entity_id = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_lang = en {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_metadata_path = /etc/keystone/saml2_idp_metadata.xml {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.975922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_organization_display_name = OpenStack SAML Identity Provider {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_organization_name = SAML Identity Provider {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_organization_url = https://example.com/ {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.976984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.idp_sso_endpoint = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.keyfile = /etc/keystone/ssl/private/signing_key.pem {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.relay_state_prefix = ss:mem: {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] saml.xmlsec1_binary = xmlsec1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.977944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.change_password_upon_first_use = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.disable_user_account_days_inactive = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.lockout_duration = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.lockout_failure_attempts = 2 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.978923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.minimum_password_age = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.979076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.password_expires_days = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.979154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.979315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.password_regex = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.979428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.password_regex_description = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] security_compliance.unique_last_password_count = 2 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] shadow_users.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.allow_expired_window = 172800 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.980992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.allow_rescope_scoped_token = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.cache_on_issue = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.981948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.expiration = 3600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.provider = fernet {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] token.revoke_by_id = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] tokenless_auth.issuer_attribute = SSL_CLIENT_I_DN {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.982933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] tokenless_auth.protocol = x509 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] tokenless_auth.trusted_issuer = [] {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] totp.included_previous_windows = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] trust.allow_redelegation = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.983963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] trust.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] trust.max_redelegation_count = 3 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] unified_limit.cache_time = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] unified_limit.caching = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] unified_limit.driver = sql {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.984914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] unified_limit.enforcement_model = flat {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.connection = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] unified_limit.list_limit = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] wsgi.debug_middleware = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.985882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.backend = sqlalchemy {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.connection = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.connection_debug = 0 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.connection_parameters = {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.connection_recycle_time = 3600 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.986998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.connection_trace = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.db_inc_retry_interval = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.db_max_retries = 20 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.db_max_retry_interval = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.987943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.db_retry_interval = 1 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.max_overflow = 50 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.max_pool_size = 5 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.max_retries = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.988962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.mysql_enable_ndb = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.mysql_sql_mode = TRADITIONAL {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.pool_timeout = None {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80233) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 24 00:46:21.989604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.retry_interval = 10 {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.989810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.slave_connection = **** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.990058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.sqlite_synchronous = True {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.990296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] database.use_db_reconnect = False {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2609}} Jul 24 00:46:21.990467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG uwsgi [-] ******************************************************************************** {{(pid=80232) log_opt_values /usr/local/lib/python3.8/dist-packages/oslo_config/cfg.py:2613}} Jul 24 00:46:22.020616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.020838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.021068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.022133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.022365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_user_delete_callback` subscribed to event `identity.user.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.022587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.application_credential.core.Manager._delete_app_creds_on_assignment_removal` subscribed to event `identity.remove_application_credentials_for_user.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.029492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.031232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.assignment.core.Manager._delete_domain_assignments` subscribed to event `identity.domain.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.059052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.059272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.060275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.project.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.060505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.catalog.core.Manager._on_project_or_endpoint_delete` subscribed to event `identity.endpoint.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.201852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.202145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.federation.core.Manager._cleanup_identity_provider` subscribed to event `identity.domain_deleted.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.251469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.251696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.252243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._domain_deleted` subscribed to event `identity.domain_deleted.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.252476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.identity.core.Manager._unset_default_project` subscribed to event `identity.project.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.316687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.316827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._trust_callback` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.316906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._consumer_callback` subscribed to event `identity.OS-OAUTH1:consumer.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._project_callback` subscribed to event `identity.project.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.user.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.317874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.revoke.core.Manager._user_callback` subscribed to event `identity.persist_revocation_event_for_user.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.342933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343767 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.343974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.344058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.user.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.344182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.344265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.domain.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.344401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.project.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.344617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.receipt.provider.Manager._drop_receipt_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.356031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.356425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.trust.core.Manager._on_user_delete` subscribed to event `identity.user.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.368651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.368749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.OS-TRUST:trust.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.368886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.368985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.deleted`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.user.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.domain.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.project.disabled`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.369970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80233) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.370061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [-] Callback: `keystone.token.provider.Manager._drop_token_cache` subscribed to event `identity.invalidate_token_cache.internal`. {{(pid=80232) register_event_callback /opt/stack/keystone/keystone/notifications.py:295}} Jul 24 00:46:22.387348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.387551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 24 00:46:22.387797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.387881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.388007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 24 00:46:22.388106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 24 00:46:22.388307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.388390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/domains' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.388521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding additional resource route (alternate) to API auth: ['/OS-FEDERATION/domains' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:433}} Jul 24 00:46:22.388602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.388789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/system' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.388873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.389034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.389114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/catalog' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.389287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens/OS-PKI/revoked' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.389541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth: ['/auth/tokens' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.400133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.400403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.400665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.400753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.400939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.401042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/saml2/ecp' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.401213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/websso/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.402043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API auth/OS-FEDERATION: ['/auth/OS-FEDERATION/identity_providers//protocols//websso' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.409889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.410266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API credentials for `MethodViewType` (API Prefix: ) [/credentials, /credentials/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.413184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.415091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.416580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API domains for `MethodViewType` (API Prefix: ) [/domains, /domains/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.416580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.416767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.416767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.416902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.417632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.417764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//config//' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config/default' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config//default' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains/config///default' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//users//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.418949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API domains: ['/domains//groups//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.443087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.443642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API ec2tokens: ['/ec2tokens' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.444692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.444968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.445256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API endpoints for `MethodViewType` (API Prefix: ) [/endpoints, /endpoints/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.445532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API endpoints: ['/endpoints//OS-ENDPOINT-POLICY/policy' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.451040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.451314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.451591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.451675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API groups for `MethodViewType` (API Prefix: ) [/groups, /groups/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.451916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.452152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API groups: ['/groups//users/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.458838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.459113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.459441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API limits for `MethodViewType` (API Prefix: ) [/limits, /limits/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.459720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API limits: ['/limits/model' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.463698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.464639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.464906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-EP-FILTER for `MethodViewType` (API Prefix: /OS-EP-FILTER) [/endpoint_groups, /endpoint_groups/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.465069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoints//projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoints' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.465899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/projects//endpoint_groups' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//endpoints' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.466844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-EP-FILTER: ['/endpoint_groups//projects/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.485316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.485505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/saml2/metadata' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.485590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.485780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-FEDERATION: ['/identity_providers//protocols//auth' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.490202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.491347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API identity_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/identity_providers, /identity_providers/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.493847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.494141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.495019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.495340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API protocols: ['/OS-FEDERATION/identity_providers//protocols' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.499417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.500383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API mappings for `MethodViewType` (API Prefix: /OS-FEDERATION) [/mappings, /mappings/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.503054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.503993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API service_providers for `MethodViewType` (API Prefix: /OS-FEDERATION) [/service_providers, /service_providers/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.506825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.507119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.507405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.507690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.507796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles//inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.507962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.508093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//groups//roles/inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.508236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.508386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles//inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.508681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/domains//users//roles/inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.508958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//users//roles//inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.509236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-INHERIT: ['/projects//groups//roles//inherited_to_projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.528253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.528613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.528877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.529148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.529246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API OS-OAUTH1 for `MethodViewType` (API Prefix: /OS-OAUTH1) [/consumers, /consumers/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.529578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/request_token' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.529847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/access_token' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.530117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-OAUTH1: ['/authorize/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.537039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.538072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API events: ['/events' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.538739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.539033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.539797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/ca' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.540068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API OS-SIMPLE-CERT: ['/OS-SIMPLE-CERT/certificates' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.542180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.542459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.542735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.543036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.543229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API policy for `MethodViewType` (API Prefix: ) [/policies, /policies/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.543314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.543508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.543788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/endpoints/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.544070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.544346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API policy: ['/policies//OS-ENDPOINT-POLICY/services//regions/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.556910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.557185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.557462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.557753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API projects for `MethodViewType` (API Prefix: ) [/projects, /projects/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.557920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//tags/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.558994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//users//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.559287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.559565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API projects: ['/projects//groups//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.578720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.579859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API regions for `MethodViewType` (API Prefix: ) [/regions, /regions/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.581986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.583159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API registered_limit for `MethodViewType` (API Prefix: ) [/registered_limits, /registered_limits/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.585291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.586481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_assignments: ['/role_assignments' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.586909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.588127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API role_inferences: ['/role_inferences' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.588533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.588805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.589628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.590320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API roles for `MethodViewType` (API Prefix: ) [/roles, /roles/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.590592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.590908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API roles: ['/roles//implies/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.597083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.598339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API s3tokens: ['/s3tokens' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.598854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.599986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API services for `MethodViewType` (API Prefix: ) [/services, /services/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.602043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.602330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.602612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.602911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.603218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.603509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/users//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.603793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.604068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API system: ['/system/groups//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.612099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.612379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.612653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.613359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API trusts for `MethodViewType` (API Prefix: /OS-TRUST) [/trusts, /trusts/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.613641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.613920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API trusts: ['/trusts//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.620501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80233) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.620768 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.621039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.621305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.621570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.621840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.621933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding standard routes to API users for `MethodViewType` (API Prefix: ) [/users, /users/] {{(pid=80232) _add_resources /opt/stack/keystone/keystone/server/flask/common.py:343}} Jul 24 00:46:22.622108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//password' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//groups' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//projects' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.622939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//credentials/OS-EC2/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.623983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80233) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.624075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.624311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//OS-OAUTH1/access_tokens//roles/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.624595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.624840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//application_credentials/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.625103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.625368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.common [-] Adding resource routes to API users: ['/users//access_rules/' {}] {{(pid=80232) _add_mapped_resources /opt/stack/keystone/keystone/server/flask/common.py:417}} Jul 24 00:46:22.662995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55628393a9c0 pid: 80233 (default app) Jul 24 00:46:22.664711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WSGI app 0 (mountpoint='') ready in 6 seconds on interpreter 0x55628393a9c0 pid: 80232 (default app) Jul 24 00:46:22.666263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47cd4dd-dbfe-489c-9b7d-af00b1704fb9 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:22.666556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47cd4dd-dbfe-489c-9b7d-af00b1704fb9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:22.666868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47cd4dd-dbfe-489c-9b7d-af00b1704fb9 None None] PATH_INFO: `/v3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:22.671329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1/1] 10.222.0.18 () {50 vars in 791 bytes} [Sun Jul 24 00:46:22 2022] GET /identity/v3/ => generated 255 bytes in 8 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:30.303450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:30.304026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:30.304026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:30.308404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1/2] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:30 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:30.313964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:30.314380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:30.314520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:30.383913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: INFO dbcounter [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] Registered counter for database keystone Jul 24 00:46:30.405546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG oslo_db.sqlalchemy.engines [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80233) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 24 00:46:30.411903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writer thread running {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 24 00:46:30.591168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG passlib.handlers.bcrypt [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80233) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 24 00:46:30.591504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG passlib.handlers.bcrypt [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80233) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 24 00:46:30.645770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:30.647414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:30.768645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 2/3] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:30 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 456 msecs (HTTP/1.1 201) 6 headers in 384 bytes (1 switches on core 0) Jul 24 00:46:30.966661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:31.036856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: INFO dbcounter [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] Registered counter for database keystone Jul 24 00:46:31.056167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG oslo_db.sqlalchemy.engines [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_ENGINE_SUBSTITUTION {{(pid=80232) _check_effective_sql_mode /usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/engines.py:314}} Jul 24 00:46:31.063183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writer thread running {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:99}} Jul 24 00:46:31.182560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9efb25f-38c7-4342-8ec1-e2c00974170d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:31.187107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:31.187768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:31.188053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:31.188330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:31.196394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:31.248325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING keystone.common.rbac_enforcer.enforcer [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 24 00:46:31.248602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0b42e58-a83d-4164-869a-23280cf11ff5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:31.258101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 2/4] 10.222.0.18 () {58 vars in 1324 bytes} [Sun Jul 24 00:46:30 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 292 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 00:46:31.264215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:31.279369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0bd01c7-6fec-4e24-9a54-327e8e611de4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:31.283774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:31.284900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:31.285184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:31.285461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] PATH_INFO: `/v3/projects/17ae0b4181524f419070ad7610eaf309` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:31.289531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] RBAC: Authorizing `identity:get_project(project_id=17ae0b4181524f419070ad7610eaf309)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:31.340339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING keystone.common.rbac_enforcer.enforcer [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] Deprecated policy rules found. Use oslopolicy-policy-generator and oslopolicy-policy-upgrade to detect and resolve deprecated policies in your configuration. Jul 24 00:46:31.340896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0b80d50-6b55-46fd-899c-b502690f0f57 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:31.345181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 3/5] 10.222.0.18 () {56 vars in 1207 bytes} [Sun Jul 24 00:46:31 2022] GET /identity/v3/projects/17ae0b4181524f419070ad7610eaf309 => generated 344 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:33.330030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2132253-84c4-4408-8b51-7d9aceef70dc None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:33.330570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2132253-84c4-4408-8b51-7d9aceef70dc None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:33.330570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2132253-84c4-4408-8b51-7d9aceef70dc None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:33.334722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 3/6] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:33 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:33.340866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:33.341267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:33.341419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:33.360354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:33.361857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:33.369496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 4/7] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:33 2022] POST /identity/v3/auth/tokens => generated 914 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 384 bytes (2 switches on core 0) Jul 24 00:46:33.391425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2132253-84c4-4408-8b51-7d9aceef70dc None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:33.403931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2132253-84c4-4408-8b51-7d9aceef70dc None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:33.408084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:33.408714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:33.409001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:33.409280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:33.417781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:33.419625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31322cd0-9ef2-4fd7-94d0-04f9a9863a45 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:33.428600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 4/8] 10.222.0.18 () {58 vars in 1324 bytes} [Sun Jul 24 00:46:33 2022] GET /identity/v3/auth/tokens => generated 914 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 00:46:33.434712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:33.443982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2aa0c43c-9fe9-4643-9023-1aa7ee8647e7 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:33.448438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:33.449299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:33.449584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:33.449864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:33.451291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:33.453128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0564c0dc-a916-4525-8668-2db07eef9d4c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:33.454693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 5/9] 10.222.0.18 () {56 vars in 1198 bytes} [Sun Jul 24 00:46:33 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 20 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:35.533467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6a61f-6aef-4211-8ca0-28cd34600f16 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.533989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6a61f-6aef-4211-8ca0-28cd34600f16 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.534080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9d6a61f-6aef-4211-8ca0-28cd34600f16 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.535419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 5/10] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:35.540468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.540841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.541026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.554281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5c50f-687b-4914-a31a-2a8fb1627012 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.554576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5c50f-687b-4914-a31a-2a8fb1627012 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.554901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9b5c50f-687b-4914-a31a-2a8fb1627012 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.556240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 6/11] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:35.559140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:35.560589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:35.562326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.562624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.562942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.602327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG passlib.handlers.bcrypt [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] detected 'bcrypt' backend, version '3.2.2' {{(pid=80232) _load_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:625}} Jul 24 00:46:35.602755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG passlib.handlers.bcrypt [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] 'bcrypt' backend lacks $2$ support, enabling workaround {{(pid=80232) _finalize_backend_mixin /usr/local/lib/python3.8/dist-packages/passlib/handlers/bcrypt.py:406}} Jul 24 00:46:35.642129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 6/12] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:35.648442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:35.650066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:35.665509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:35.691220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:35.751349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 7/13] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:35.773199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-108d13f5-fc72-42fa-9c13-e587345424ba None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:35.773502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:35.808247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:35.809050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.809340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.809621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.821263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:35.832717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:35.833799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:35.834693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 7/14] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:46:35 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:46:35.840650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:35.873915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8d56b2f-72d4-4a1d-b5ab-7f48587c2b78 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:35.876188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:35.911724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:35.912615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.912905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.913187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.922207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:35.940941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2770b9-27c5-4c0e-8d22-b96f475ea546 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:35.975566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:35.976237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:35.976522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:35.976800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:35.977554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:35.977936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:35.979679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:35.980838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:35.982378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:35.991785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 8/15] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:35 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:35.997588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:36.079655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 8/16] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:35 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:46:36.085418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:36.088128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:36.098292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0539638-1392-4b0d-b919-27a1d435e76d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:36.131816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:36.132697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:36.132984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:36.133259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:36.134564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:36.136763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:36.138515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 9/17] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:35 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:36.144033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:36.158921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:36.183441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d14aec84-9c9c-4596-8bd1-106c8338e973 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:36.217638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:36.218438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:36.218725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:36.219036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:36.237144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:36.238621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:36.243808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2b284e2-9058-4205-bb90-8d9ef49e0f85 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:46:36.261921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:36.262790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:36.263598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:46:36.263598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:46:36.263598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:46:36.263598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:36.263598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 9/18] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:46:36.268823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:36.277393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:36.278194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:36.278472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:36.278742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:36.279946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:36.281414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:36.283040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 10/19] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:36 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:36.288580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:36.325204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:36.368763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:36.370480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33aa1475-4de8-4e67-a842-389e13d3a56a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:36.389636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d6222a-6d15-4e87-86ab-f6e36640727e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:36.407667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:36.408339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:36.408628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:36.408911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:36.409637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:36.409948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:36.411396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-728f71f4-dd89-4361-9229-0946923a2298 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:36.425601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:36.426267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:36.426552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:36.426859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:36.427696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.domain.id=default, user.id=f00ceffcf1cf4b95908b3794e7b9b31d) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:36.428008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:36.429540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 10/20] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:46:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:36.429657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15838941-bfcd-46f7-ba3f-22ee39330b95 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:36.437382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 11/21] 10.222.0.18 () {56 vars in 1322 bytes} [Sun Jul 24 00:46:36 2022] GET /identity/v3/role_assignments?user.id=f00ceffcf1cf4b95908b3794e7b9b31d&scope.domain.id=default&role.id=75e91b1e89304179904cf7644ef0a984 => generated 233 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:38.555927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f9c25-9572-46a8-a700-4ecd5cee9dd9 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:38.556457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f9c25-9572-46a8-a700-4ecd5cee9dd9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:38.556539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0f9c25-9572-46a8-a700-4ecd5cee9dd9 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:38.558467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 11/22] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:38.563973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:38.564367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:38.564633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:38.586889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:38.588097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:38.676235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 12/23] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:38 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:38.709068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f9c25-9572-46a8-a700-4ecd5cee9dd9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:38.838447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf0f9c25-9572-46a8-a700-4ecd5cee9dd9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:38.879458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:38.883905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:38.884922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:38.886048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:38.886615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:38.901381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:38.910555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:38.911413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:38.912212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:38.915883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 12/24] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:46:38 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:38.922113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.021677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:39.053901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c92b21c6-aaa7-4c62-bd49-4dca9ac5617d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.097544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.098231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.098517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.098812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.099582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:39.099890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:39.101315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cdb964-395d-47f7-a31e-3b7acadcf85e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:39.113991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 13/25] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:38 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:39.120514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.151753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9fe977-7c23-42a6-a02f-47497747f369 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.152054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9fe977-7c23-42a6-a02f-47497747f369 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.152327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb9fe977-7c23-42a6-a02f-47497747f369 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.153713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 14/26] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:39 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:39.158976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d794fd56-1068-4312-9141-5acbbbb05568 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.159268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d794fd56-1068-4312-9141-5acbbbb05568 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.159553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d794fd56-1068-4312-9141-5acbbbb05568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.191889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d794fd56-1068-4312-9141-5acbbbb05568 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:39.193470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d794fd56-1068-4312-9141-5acbbbb05568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:39.270852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41208d8-b50f-4404-aaaf-a837d1c4c6f2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.319781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.320581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.320873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.321218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.327883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:39.334812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8da4bb1d-0850-404c-8305-4c03778fe8ce None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:39.334812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 13/27] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:39 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:39.334812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5f94-03e6-4a67-a9f7-b10635d06590 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.334812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5f94-03e6-4a67-a9f7-b10635d06590 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.334812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed5f94-03e6-4a67-a9f7-b10635d06590 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.335189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 14/28] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:39.337262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed5f94-03e6-4a67-a9f7-b10635d06590 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.404551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 15/29] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:39.405695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.405993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.406349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.415129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:39.437944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:39.439203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:39.536714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed5f94-03e6-4a67-a9f7-b10635d06590 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.552049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:39.566800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 16/30] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:39.566800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.595430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.596778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.597232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.597647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.599163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:39.600852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80056abe-cb33-4656-a083-757f9a7fb433 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:39.603122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 15/31] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:39 2022] GET /identity/v3/domains/default => generated 199 bytes in 266 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:39.604700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb791bb-0248-4194-bc76-d71e70801330 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.605139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb791bb-0248-4194-bc76-d71e70801330 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.605658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb791bb-0248-4194-bc76-d71e70801330 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.611162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 16/32] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:39 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:39.612261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb791bb-0248-4194-bc76-d71e70801330 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.674980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1ec3da7-6955-4185-a4f3-1d6fd58194bc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.711994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.713097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.713392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.713728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.714798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:39.716382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd8e17d0-9cfb-42d2-a524-6c1e157ad3fc None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:39.727445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb791bb-0248-4194-bc76-d71e70801330 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.742887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:39.768085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.768882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.769171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.769451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.780982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:39.782541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:46:39.801355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:39.802186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:39.803032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:46:39.803032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:46:39.803032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:46:39.803032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:39.803032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 17/33] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:39.808478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:39.878557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 17/34] 10.222.0.18 () {60 vars in 1131 bytes} [Sun Jul 24 00:46:39 2022] POST /identity/v3/roles => generated 219 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:39.881515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.882062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.882584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:39.927988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:39.929588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:39.955296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae037575-c38d-412d-a04a-05437ba5f559 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:39.992629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:39.993674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:39.994009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:39.994296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] PATH_INFO: `/v3/domains/default/users/f00ceffcf1cf4b95908b3794e7b9b31d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.001247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] RBAC: Authorizing `identity:create_grant(domain_id=default, user_id=f00ceffcf1cf4b95908b3794e7b9b31d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:40.001247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:40.043928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:40.067213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 18/35] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:39 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:40.068207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:40.137809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 18/36] 10.222.0.18 () {58 vars in 1361 bytes} [Sun Jul 24 00:46:39 2022] PUT /identity/v3/domains/default/users/f00ceffcf1cf4b95908b3794e7b9b31d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 335 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:46:40.139009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:40.173560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:40.210500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39b0854-8a5c-4e8f-b896-b30e47b57213 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:40.260905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:40.261873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.262163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.262439 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.263424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:40.263743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:40.265333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a27e42bd-eac2-480a-a8bb-422d7f652d63 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:40.280884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b407b3fd-8fff-4ed8-9d2b-32a2fe4ab4db None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:40.299370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 19/37] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:46:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 232 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:40.301877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e084-832b-431b-9177-5a01ae2bcb07 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.302209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e084-832b-431b-9177-5a01ae2bcb07 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.303017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3366e084-832b-431b-9177-5a01ae2bcb07 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.304896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 20/38] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:40 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:40.308299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366e084-832b-431b-9177-5a01ae2bcb07 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:40.340737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:40.341438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.341731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.342017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.342872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:40.344514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec65b8bf-97d8-45aa-ab57-fe86e6ad5d72 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:40.370831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 19/39] 10.222.0.18 () {60 vars in 1131 bytes} [Sun Jul 24 00:46:40 2022] POST /identity/v3/roles => generated 225 bytes in 232 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:40.372216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.372482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:40.372584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.372783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.409069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:40.410631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:40.474182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3366e084-832b-431b-9177-5a01ae2bcb07 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:40.556338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:40.558558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.560731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.561240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.562910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:40.565462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27e8b6e3-663a-4701-b8c5-d9411069753f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:40.582195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 20/40] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:40.608701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:40.641033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:40.713375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 21/41] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:46:40 2022] POST /identity/v3/projects => generated 301 bytes in 402 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:40.713375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca6b76-8211-41f7-b0f2-e9b9cc998698 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.713375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca6b76-8211-41f7-b0f2-e9b9cc998698 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.713375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca6b76-8211-41f7-b0f2-e9b9cc998698 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.713375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 22/42] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:40.718950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.719245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.719518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.741420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:40.742793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:40.749893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d407b364-fcfa-4ce4-8bc4-d8d05e793942 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:40.798570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:40.799493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:40.800352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:40.800712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:40.802000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:40.803880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:40.830903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 21/43] 10.222.0.18 () {60 vars in 1131 bytes} [Sun Jul 24 00:46:40 2022] POST /identity/v3/roles => generated 223 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:46:40.837946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 23/44] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:40.878326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:40.970805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:41.002331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8667e4cb-7aa6-4a42-838e-e7a6f4400a06 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:41.038622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:41.039461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:41.039801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:41.040089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:41.041280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:41.043063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69a33bbf-8208-4751-8222-ec7e1f580067 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:41.044930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 22/45] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:40 2022] GET /identity/v3/domains/default => generated 199 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:41.050339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:41.081527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:41.154356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d27b47c-43ec-4820-bb37-2a1597fb42a1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:41.188028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:41.188693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:41.188985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:41.189270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:41.189997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:41.191516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e38487c-8ff6-4137-b959-58aed1b7566c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:41.216309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 24/46] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:46:41 2022] POST /identity/v3/projects => generated 312 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:42.292741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0f93a0-e3ad-49f4-8b11-e3fb4078f05d None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:42.293256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0f93a0-e3ad-49f4-8b11-e3fb4078f05d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:42.293359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba0f93a0-e3ad-49f4-8b11-e3fb4078f05d None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:42.294881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 23/47] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:42.299778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:42.300175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:42.300458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:42.317699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:42.318925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:42.387642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 25/48] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:42.410344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0f93a0-e3ad-49f4-8b11-e3fb4078f05d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:42.509114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba0f93a0-e3ad-49f4-8b11-e3fb4078f05d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:42.540758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:42.541580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:42.541998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:42.542281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:42.552259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:42.560638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:42.561473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:42.562556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 24/49] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:46:42 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:42.567606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:42.579009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:42.671039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4762838c-65a9-47cc-890b-bce16b114112 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:42.705277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:42.705951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:42.706237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:42.706516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:42.707305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:42.707616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:42.709009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:42.718063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 26/50] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:42 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:42.723614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:42.727072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:42.825075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2acc7158-bcc8-42ac-a7a8-ab6aef85993d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:42.858669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:42.859593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:42.859879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:42.860159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:42.861339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:42.862790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:42.864578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 25/51] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:42 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:42.870485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:42.939419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:42.969116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686340ed-4531-45b0-bf79-5aa2c96b0967 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:43.002780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:43.003600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.003889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.004169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.005341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:43.006830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9166e7f4-ed44-4b88-9d84-0f51c661e17d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:43.008460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 27/52] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:42 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:43.015263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:43.083476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:43.114802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fdb0cd6-3cf6-4f98-920f-b559af66fea1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:43.252993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:43.253654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.253948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.254232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.255109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.domain.id=default, user.id=f00ceffcf1cf4b95908b3794e7b9b31d) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:43.255428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:43.256874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93fd6785-b953-4407-ba49-4dabb22bd60e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:43.265275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 26/53] 10.222.0.18 () {56 vars in 1322 bytes} [Sun Jul 24 00:46:43 2022] GET /identity/v3/role_assignments?user.id=f00ceffcf1cf4b95908b3794e7b9b31d&scope.domain.id=default&role.id=75e91b1e89304179904cf7644ef0a984 => generated 530 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:43.265676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d88bd00-6559-4f44-92b6-a502415959c9 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.266034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d88bd00-6559-4f44-92b6-a502415959c9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.266441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d88bd00-6559-4f44-92b6-a502415959c9 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.267958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 28/54] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:43.273438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.273733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.274019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.292949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:43.294189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:43.366007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 27/55] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:43 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:43.389647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d88bd00-6559-4f44-92b6-a502415959c9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:43.500327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d88bd00-6559-4f44-92b6-a502415959c9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:43.533368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:43.534165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.534449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.534725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.535931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:43.537385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5827cef8-287c-4b3d-bd61-397bf9809bdc None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:43.539039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 29/56] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:43.544989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:43.603761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:43.640090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db8e29-b9a8-4f26-9717-3351b6f2bb76 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:43.672459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:43.673117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:43.673403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:43.673681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:43.674392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:43.675859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7e28ae2-b32b-4b86-9d4b-46f0b8b4e110 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:43.701308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 28/57] 10.222.0.18 () {60 vars in 1140 bytes} [Sun Jul 24 00:46:43 2022] POST /identity/v3/projects => generated 298 bytes in 157 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:46:45.724151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108f646-6f5a-4b44-9a3b-c2e2bc589279 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:45.724645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108f646-6f5a-4b44-9a3b-c2e2bc589279 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:45.724645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8108f646-6f5a-4b44-9a3b-c2e2bc589279 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:45.726007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 30/58] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:45.731108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:45.731509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:45.731716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:45.748264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:45.749458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:45.814494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 29/59] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:45.839311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8108f646-6f5a-4b44-9a3b-c2e2bc589279 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:45.849963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:45.937865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8108f646-6f5a-4b44-9a3b-c2e2bc589279 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:45.972219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:45.973004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:45.973284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:45.973559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:45.974816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:45.976211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16521bc0-bfd2-4459-bed8-307afb1c0d33 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:45.977808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 31/60] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:45 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:45.983653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:46.028801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:46.080101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ee2e7a-4216-43d9-9cf6-2aa7db2de479 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:46.113389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:46.114058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:46.114345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:46.114624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:46.115377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:46.116828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be5e4b61-f738-43f5-9f18-ddfa3ba2051f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:46.143134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 30/61] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:46:45 2022] POST /identity/v3/users => generated 275 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:49.186376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be010735-523f-4674-bc45-062b0ac28220 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.187353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be010735-523f-4674-bc45-062b0ac28220 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.187811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be010735-523f-4674-bc45-062b0ac28220 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.189642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 32/62] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:49.195291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e97d81-10d3-4efd-85df-d806dae0d303 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.196059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e97d81-10d3-4efd-85df-d806dae0d303 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.196491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7e97d81-10d3-4efd-85df-d806dae0d303 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.223819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a7e97d81-10d3-4efd-85df-d806dae0d303 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:49.225269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a7e97d81-10d3-4efd-85df-d806dae0d303 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:49.316354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 31/63] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:49.351916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be010735-523f-4674-bc45-062b0ac28220 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:49.454699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:49.488424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be010735-523f-4674-bc45-062b0ac28220 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:49.537985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:49.538875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.539168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.539503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.545386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758185ed-6050-4731-8bde-77fb476942c4 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.545862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758185ed-6050-4731-8bde-77fb476942c4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.546285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-758185ed-6050-4731-8bde-77fb476942c4 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.548062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 32/64] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:49 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:49.553220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546d556-c75f-483f-9d29-f21f38d301b4 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.553700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546d556-c75f-483f-9d29-f21f38d301b4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.554122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4546d556-c75f-483f-9d29-f21f38d301b4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.555510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:49.564412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:49.565290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:49.566040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 33/65] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:49 2022] GET /identity/v3/roles/member => generated 84 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:49.572554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:49.599796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4546d556-c75f-483f-9d29-f21f38d301b4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:49.601111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4546d556-c75f-483f-9d29-f21f38d301b4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:49.636649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:49.687034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62854917-5f0f-47c5-bbc1-f2e41ae405b0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:49.691214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 33/66] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:49.698790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1673bbc-0f71-4955-8227-bd4d06a79dd9 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.699460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1673bbc-0f71-4955-8227-bd4d06a79dd9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.699907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1673bbc-0f71-4955-8227-bd4d06a79dd9 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.701872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 34/67] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:49 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:49.707058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780540b5-1613-408a-a74d-4a5d687a33c6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.707521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780540b5-1613-408a-a74d-4a5d687a33c6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.707942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780540b5-1613-408a-a74d-4a5d687a33c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.730176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-780540b5-1613-408a-a74d-4a5d687a33c6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:49.731636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-780540b5-1613-408a-a74d-4a5d687a33c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:49.733465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:49.734232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.734593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.734979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.735808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:49.736318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:49.737902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:49.740673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:49.749517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 34/68] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:49 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:49.750611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:49.811946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 35/69] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:49.813524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76328fad-b55a-4991-8d72-8401e8907f76 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.813815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76328fad-b55a-4991-8d72-8401e8907f76 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.814094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76328fad-b55a-4991-8d72-8401e8907f76 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.815390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 36/70] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:49 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:49.816297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76328fad-b55a-4991-8d72-8401e8907f76 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:49.870562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bda2d2aa-1f79-4534-a6cf-a082c6d3b42e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:49.907007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:49.907822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.908116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.908399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.919605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:49.926380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-6d66c7c7-9c84-4fae-bcb4-2f72853e4d26 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:49.927421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:49.928479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:49.929461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 35/71] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:46:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:49.930238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.930238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.930383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:49.947023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76328fad-b55a-4991-8d72-8401e8907f76 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:49.947459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:49.948784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:49.985568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:49.986693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:49.987011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:49.987292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.003080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.004567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.006289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 37/72] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:49 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.006858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.011662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.023485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 36/73] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:50.024684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.115538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c100df2e-eadb-490d-867c-c5f604e3d65a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.133507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13d15a5e-7d2d-4f3d-9d37-6155ee73b851 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.149279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.150099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.150391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.150679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.161250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:50.168121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.168790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.169076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.169353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:50.169963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:50.170973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:46:50.172005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:50.173054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:50.173054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.173213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 38/74] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:50.173290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bcc28e9f-1c4f-4c10-8ec2-db3d83fcfaea None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.173366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.181422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 37/75] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.182796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaef26f-94ea-4fec-b957-986d804c85f2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.183095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaef26f-94ea-4fec-b957-986d804c85f2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.183376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecaef26f-94ea-4fec-b957-986d804c85f2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.186808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 38/76] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:50.187354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaef26f-94ea-4fec-b957-986d804c85f2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.188147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.240050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.277250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47b6163-d5b3-4f08-8ebf-003bd5fbeaf7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.290718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecaef26f-94ea-4fec-b957-986d804c85f2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.311416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.312075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.312366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.312652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.327870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.328675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.328959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.329237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.339755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:50.347607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:50.348443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:50.349184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 39/77] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/roles/member => generated 84 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:50.349895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.383321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.384807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.396713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.462360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d9f2d90-d41c-4325-a565-8253c32aab87 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.475150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.492955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 39/78] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:50.493851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.507268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.507953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.508243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.508524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.509327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:50.509634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.511279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56758d60-5639-45af-8bde-dbe2843083d2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.521755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 40/79] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.523074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.523371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.523655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.540418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:50.542357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:50.610726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc984bbf-36b8-4b81-a5de-2f57af27d361 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.620558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 41/80] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:50.621426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.636847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.651911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.652808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.653096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.653375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.654615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.656118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.657768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 40/81] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.658614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.747136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcbc3a6f-9009-4c63-b436-832c56ebd221 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.748741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.768924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-603412a9-7433-4746-87bd-c80dc15864e9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.786127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.786801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.787081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.787344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.788654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:50.789095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.792098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.803288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 42/82] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.804474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.808610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.809724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.810014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.810557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.811831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.813316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.815084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 41/83] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.815860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.866558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:50.915283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45843bde-8a34-40f6-ad50-fe6bee203e49 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.933901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3956e299-3852-4840-b1bb-0c0d7ef09bd7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:50.952013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.953029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.953319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.953597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.954975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.956516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.958161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 43/84] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.958951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.978734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:50.979698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:50.979983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:50.980265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:50.981475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:50.982985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:50.984644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 42/85] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/domains/default => generated 199 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:50.985421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:50.999627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.078821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c463050c-0fff-4453-99a5-6252c93867b8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.099949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-409217a1-165e-4f63-99f9-e2d708250603 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.114535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.120017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.120976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.121286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.121566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.136155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.137069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.137350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.137624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.138953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.140892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.142698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 43/86] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:51.143502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.198813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.200298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.231077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.256466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608f90b2-5bec-4c1f-b30a-0ad89182aa58 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.290629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.291347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.291727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.292010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.292852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:51.293176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.294641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.299673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 44/87] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:50 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:51.300348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 44/88] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=881cc16b876541819935dfa513a16e5d => generated 259 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:51.300571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.300956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.360354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.422870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ab2e6f9-dd23-4f47-9150-1ebc98fa895a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.426379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4dade6-a594-4fac-9ca5-480d7d9a6bb2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.457252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.468154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.468319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.468552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.468660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.468810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.469126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.469411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.469695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.470420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.472105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.496131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 45/89] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:46:51 2022] POST /identity/v3/users => generated 289 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:46:51.496861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.582449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.585187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.629873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbee8bb0-7b92-42f4-bb0f-6e8c88927015 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.632002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.680485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.683179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.684446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.685254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.738155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 45/90] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 438 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:51.739383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.786679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.788617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.788882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:51.863903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56cb4e6e-f2c7-406f-aa1d-4e63513842f4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:51.903413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:51.904300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:51.904590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:51.904874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:51.906123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:51.907630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:51.908214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 46/91] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 412 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:51.908929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.909532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 46/92] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:51.914837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:51.927474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:52.076069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:52.079608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49afdde7-4ca5-45eb-983d-f6aec5fe2d8c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:52.084498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed48292e-3405-4248-a7f2-dc84b98c4c97 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:52.138807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:52.139623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:52.139909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:52.140190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] PATH_INFO: `/v3/projects/fe820d8d7de0479598637dfa23756659` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:52.141429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] RBAC: Authorizing `identity:get_project(project_id=fe820d8d7de0479598637dfa23756659)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:52.142944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:52.145501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 47/93] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/projects/fe820d8d7de0479598637dfa23756659 => generated 312 bytes in 231 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:52.146796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:52.167510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:52.168407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:52.168692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:52.168971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:52.170224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:52.171702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:52.173581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 47/94] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:51 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 265 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:52.175228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:52.285543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:52.320485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ec865c-25b3-4f4d-82f1-cabad3aca5c3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:52.322400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f94ba31-ef7c-4d34-8f27-ef7016d141c7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:52.370053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:52.371194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:52.371668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:52.372127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:52.373179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=fe820d8d7de0479598637dfa23756659, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:52.373678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:52.375363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:52.384008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 48/95] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:52 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=fe820d8d7de0479598637dfa23756659&role.id=881cc16b876541819935dfa513a16e5d => generated 259 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:52.386220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:52.392865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:52.393837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:52.394305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:52.394763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:52.395860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=f00ceffcf1cf4b95908b3794e7b9b31d) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:52.396370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:52.398096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc2c4256-a58c-4679-8cd5-2469ca11605b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:52.428652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 48/96] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:52 2022] GET /identity/v3/role_assignments?user.id=f00ceffcf1cf4b95908b3794e7b9b31d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 283 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:52.569764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c8bb42-fca2-4f4c-a513-4a33086202f9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:52.624380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:52.625071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:52.625361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:52.625644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:52.626495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:52.626834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:52.628355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389d7eb7-9313-4aab-a68b-c16dd9656389 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:52.630493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:52.640523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 49/97] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:52 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=496aed2e00104d4480f0efe375a79409 => generated 259 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:53.647049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18501a-a2ee-4f42-93e0-789e9c9d623f None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:53.647547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18501a-a2ee-4f42-93e0-789e9c9d623f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:53.647618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f18501a-a2ee-4f42-93e0-789e9c9d623f None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:53.649002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 49/98] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:53.660217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:53.660661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:53.660834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:53.687816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:53.688958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:53.761785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 50/99] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:53 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:53.784877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f18501a-a2ee-4f42-93e0-789e9c9d623f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:53.817168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:53.900487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f18501a-a2ee-4f42-93e0-789e9c9d623f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:53.944312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:53.945052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:53.945342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:53.945624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:53.960542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:53.968617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:53.969492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:53.970233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:53.975699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 50/100] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:53 2022] GET /identity/v3/roles/member => generated 84 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:53.982153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:54.112056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-663ac147-7c7b-4da5-bc27-50e456791ae9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:54.149078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:54.163497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:54.164185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.164481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.164771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.165548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:54.165867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:54.167889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:54.180619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 51/101] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:53 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:54.187219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:54.295594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:54.305750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1858119b-6baf-4dd1-ab36-5eaa669c41ae None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:54.359514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:54.360427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.360720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.361006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.362363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:54.363778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:54.365566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 51/102] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:54 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:54.371852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:54.521531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebfccee5-c8d8-43f6-907c-9fc841544c34 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:54.565766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:54.566592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.566910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.567195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.568541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:54.570077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273661b6-6593-426f-aabd-e229fdc51958 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:54.571854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 52/103] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:54 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:54.578169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:54.654012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564626ea-1912-4d51-af45-6339d2c1d7d5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.654320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564626ea-1912-4d51-af45-6339d2c1d7d5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.654588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-564626ea-1912-4d51-af45-6339d2c1d7d5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.656073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 53/104] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:54.661079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.661624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.661624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.690239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:54.691483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:54.709905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05016d85-a1fb-4afe-9fbf-b83a35ad117b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:54.823690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:54.824551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.824827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.825112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/801a75d4248e4a86a827af56143e1da2/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.827034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 54/105] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:54.827646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:54.828333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, user_id=801a75d4248e4a86a827af56143e1da2, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:54.830060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43f3efc7-d0ca-4470-b39a-3b201f3b56d1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:54.852372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 52/106] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:46:54 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/801a75d4248e4a86a827af56143e1da2/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 275 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:46:54.866412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:54.927172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc3170-7763-4f6e-ae74-86770deffdd1 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.927784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc3170-7763-4f6e-ae74-86770deffdd1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.928277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9bc3170-7763-4f6e-ae74-86770deffdd1 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.930060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 53/107] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:54 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:54.937670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:54.938301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:54.938790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:54.976522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:54.978156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:55.019193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:55.025333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29b607a0-89ce-4b49-a4c2-e57bfeebf06d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:55.068334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:55.069747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.070433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.071097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.072582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:55.075209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:55.077827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 55/108] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:54 2022] GET /identity/v3/domains/default => generated 199 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:55.085446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:55.124410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 54/109] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:54 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:55.154511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:55.172754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:55.204649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37944c65-65d0-4e32-8c54-7271c81a2029 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:55.250795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:55.250795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.250795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.250795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.251193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:55.251193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:55.287479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 56/110] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:46:55 2022] POST /identity/v3/projects => generated 302 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:46:55.374851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a07e458e-8611-4eef-b581-b71d56b11f30 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:55.459525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:55.460360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.460641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.460916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.475696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-34e4c261-31be-4c3a-a246-2c9e6e9be428 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:55.484486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:55.485393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:55.486623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 55/111] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:55 2022] GET /identity/v3/roles/reader => generated 84 bytes in 314 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:55.491768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:55.556412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de1b3a-306c-4102-a189-0e72f87f7420 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.556722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de1b3a-306c-4102-a189-0e72f87f7420 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.557105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68de1b3a-306c-4102-a189-0e72f87f7420 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.558509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 56/112] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:55.563951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab6f33-2740-4b87-a24e-059405e3c65c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.564248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab6f33-2740-4b87-a24e-059405e3c65c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.564538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8ab6f33-2740-4b87-a24e-059405e3c65c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.599949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c8ab6f33-2740-4b87-a24e-059405e3c65c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:55.601181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c8ab6f33-2740-4b87-a24e-059405e3c65c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:55.608755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:55.622610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4e277ea-cfe8-4dfd-9680-8ffedbb385c0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:55.634479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:55.662492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:55.663267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.663518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.663803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.664693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:55.665261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:55.666779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:55.676188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 57/113] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:55 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:55.685201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:55.769111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 57/114] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:55.789470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d223baa-b860-4d44-94a6-8a68f9d668cd None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.789765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d223baa-b860-4d44-94a6-8a68f9d668cd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.790041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d223baa-b860-4d44-94a6-8a68f9d668cd None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.791798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 58/115] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:46:55.794350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e60631f3-2d9e-496f-abe3-a0cbf4dcac23 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:55.797178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.797471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.797758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.834588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:55.835817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:55.837019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:55.838863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:55.839209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:55.839498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] PATH_INFO: `/v3/users/8a7efcb9024144239e6369dc3c16e746` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:55.859678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] RBAC: Authorizing `identity:get_user(user_id=8a7efcb9024144239e6369dc3c16e746)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:55.862012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:55.865516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 58/116] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:55 2022] GET /identity/v3/users/8a7efcb9024144239e6369dc3c16e746 => generated 289 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:55.866300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:55.899850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:55.930208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 59/117] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:55.930879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.007660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.040937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd786f59-6099-42e0-9677-3e3feb16115f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.041122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dc106da-bbb1-4f8b-9af4-b78bbda0a428 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.077535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.078700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.079213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.079702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.084492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.085160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.085453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.085729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.098646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:56.110012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:56.111001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:56.112192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 59/118] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:55 2022] GET /identity/v3/roles/member => generated 84 bytes in 246 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:56.112807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.170001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.172153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46e98522-017c-4f06-9679-d1942b6e32c9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.236928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.254298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4e13a6f-c839-4892-b28c-a8435309271b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.282060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 60/119] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:55 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 352 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:56.283655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbbc24-c933-47dd-a5d2-1fdf2b588b8d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.283944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbbc24-c933-47dd-a5d2-1fdf2b588b8d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.284218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4abbbc24-c933-47dd-a5d2-1fdf2b588b8d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.285824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 61/120] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:56 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:56.286326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abbbc24-c933-47dd-a5d2-1fdf2b588b8d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.287314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.300856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.302403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.303519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.304011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.319189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:56.330898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:56.331766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:46:56.332539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:56.334667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 60/121] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 222 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:56.335613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.422061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4abbbc24-c933-47dd-a5d2-1fdf2b588b8d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.467126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.467803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.468092 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.468370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.468465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732706c4-e800-40fd-ab7e-b04855f02365 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.469122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:56.469433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.472096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df942e34-90fa-4813-984a-ecf5d138b7b4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.482431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 62/122] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:56.483687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.483974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.484355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.505848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:56.507120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:56.518111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.519574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.520028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.520317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.522367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.524608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.527934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 61/123] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:56.529058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.537601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.552810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.594670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 63/124] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:56 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:56.596120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.658248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbf8b24a-32af-483a-9863-9e9556693a6c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.705228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.705841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.706110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.706399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.707225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:56.707794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.709855 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.711624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72118879-e0ae-402a-a4cd-57c3022e7b0b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.720241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 62/125] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:56.720875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.749442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.751161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.751786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.752392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.754294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.756455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.761318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 64/126] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:56.762276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.799877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.832878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:56.862692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8826bbe-4976-4785-8e8e-2df2e7eb5b3c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.879717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c17d92-858b-4fd3-bfb4-0975867e6659 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:56.914406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.915523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.915922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.916371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.917327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=8a7efcb9024144239e6369dc3c16e746) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:56.917822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:56.919535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:56.919940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:56.920894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:56.921329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:56.921741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:56.926850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 63/127] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/role_assignments?user.id=8a7efcb9024144239e6369dc3c16e746&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=b65cce2fe3734269906540559fc82939 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:56.927985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:56.935739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:56.944383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:56.945308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:46:56.946104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:46:56.949812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 65/128] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/roles/admin => generated 83 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:56.951103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.060158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd0559d-048b-4c45-9c6a-a6473591d6b8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.070307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.103266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985c3fdc-3457-4037-8a56-0204cf7cf025 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.109969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.110906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.111200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.111483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.118270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.120642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.122665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 64/129] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:57.126248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.159241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.189006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.190413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.191380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.191910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] PATH_INFO: `/v3/projects/fe820d8d7de0479598637dfa23756659` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.193407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] RBAC: Authorizing `identity:get_project(project_id=fe820d8d7de0479598637dfa23756659)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.195244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.198228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 66/130] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:56 2022] GET /identity/v3/projects/fe820d8d7de0479598637dfa23756659 => generated 312 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:57.199523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98537507-87d9-41fb-ad57-4afa64e170d6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.301655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.302337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.302629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.302943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.303704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:57.304020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.305404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.321851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 65/131] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:57 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:57.323071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.401897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e19226d-7a7c-4a9c-ba75-9718ad8c59d5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.447161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.477519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bda40e-e6d5-45ee-9c12-b71d004ebcd2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.496531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.497424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.497716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.498002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.503553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.505069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.506851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 67/132] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:57 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 308 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:57.507630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.515220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.516964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.517257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.517539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] PATH_INFO: `/v3/projects/fe820d8d7de0479598637dfa23756659/users/801a75d4248e4a86a827af56143e1da2/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.524308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] RBAC: Authorizing `identity:create_grant(project_id=fe820d8d7de0479598637dfa23756659, user_id=801a75d4248e4a86a827af56143e1da2, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.527693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.547894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 66/133] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:46:57 2022] PUT /identity/v3/projects/fe820d8d7de0479598637dfa23756659/users/801a75d4248e4a86a827af56143e1da2/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:46:57.548853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.559403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.668106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9f364e2-497f-4d6f-9e30-0f5d7a1e0912 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.703247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.708075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.709313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.709969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.710287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/801a75d4248e4a86a827af56143e1da2/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.713244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, user_id=801a75d4248e4a86a827af56143e1da2, role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.714982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-beefeb73-876e-4ee5-ad26-86f02a96b4ed None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.717089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b319d3-9b26-4a79-bcb0-0b9322a94460 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.731445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 67/134] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:46:57 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/801a75d4248e4a86a827af56143e1da2/roles/496aed2e00104d4480f0efe375a79409 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:46:57.732887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b4dac-d3a8-4451-b2c8-935f940cb444 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.733187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b4dac-d3a8-4451-b2c8-935f940cb444 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.733515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-708b4dac-d3a8-4451-b2c8-935f940cb444 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.735150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 68/135] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:57.747809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.748109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.748392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.780638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.782069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.782622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.783137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.783969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:57.784846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.785186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:57.786847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.789142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 68/136] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:57 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 282 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:57.797178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.916897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:57.927605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffdf5c7d-c6a3-4761-8903-fd65d60c7dd2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:57.969896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:57.971171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:57.972097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:57.972584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:57.974466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:57.977088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:57.980119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 69/137] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:57 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:57.990645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:57.992798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 69/138] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 246 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:46:58.031888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.143435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54075f0a-af35-4e08-872f-c00419f1137c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:58.189775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d71e43ae-9326-4447-b0aa-0b9d0b812d64 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:58.196310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:58.197341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.197785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.198162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/f00ceffcf1cf4b95908b3794e7b9b31d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.204105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, user_id=f00ceffcf1cf4b95908b3794e7b9b31d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:58.204105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71439a2b-8f11-47b0-b7bd-53b1ac151f11 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:58.223866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 70/139] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:46:57 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/f00ceffcf1cf4b95908b3794e7b9b31d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:46:58.235191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:58.257779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:58.258624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.258954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.259240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.259805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9c32c-a27a-4f7e-8ca1-a11ce175d24b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.260323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9c32c-a27a-4f7e-8ca1-a11ce175d24b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.260840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bc9c32c-a27a-4f7e-8ca1-a11ce175d24b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.263036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 71/140] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:58 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:58.269087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.269773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.270266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.274468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:46:58.283997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:46:58.284887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:46:58.285663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 70/141] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:46:58 2022] GET /identity/v3/roles/member => generated 84 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:46:58.298807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.308965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:58.310860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:46:58.354705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:58.483001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 72/142] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:46:58.521160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.543097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1683ce-89d4-43a2-88c7-01420c40206b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:58.601431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:58.602519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.603055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.603533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.605276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:58.605834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:58.607451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:58.624239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 71/143] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:46:58 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 334 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:58.629918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.646067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4f93be6-0b20-45ad-8742-1f8f8abf5176 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:58.687972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:58.699739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:58.700556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.700912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.701114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.707499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:58.708955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:58.710605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 73/144] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:46:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:46:58.716568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.751073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afc5df46-1182-4c21-8fe6-3a05a4f6d15f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:58.753722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:58.799369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:58.800635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:58.801214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:58.801717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:58.805961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:58.809272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:58.814745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 72/145] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:46:58 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:58.824057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:58.936671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7030caa0-dc29-440e-9766-69b6ae19943d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:59.012195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:59.013221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.013711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.014270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.015339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:59.017018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:59.042838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d203dc0-758b-442e-ba19-6eeaa22195e4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:59.065032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 74/146] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:46:58 2022] POST /identity/v3/users => generated 283 bytes in 349 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:46:59.068787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:59.110493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:59.111208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.111498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.111784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.210430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:59.211979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:59.360682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 73/147] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:46:58 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 537 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:46:59.366059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:59.473622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:59.498681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10187b12-ee48-49b7-a202-3a1cbdbaf166 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:59.535915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:59.536784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.537064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.537338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.538553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:59.540053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574dc81f-a59a-4a24-8908-56e7b91f2d09 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:59.542295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 75/148] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:46:59 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:59.548165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:46:59.574149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:46:59.695423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46479278-9bbb-4062-bcda-0c9a3799dd22 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:46:59.740968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:46:59.741878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.742320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.742739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.743763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:46:59.745018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:46:59.746738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe3f012-41f6-44ea-ba26-d678ed0ba36c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:46:59.759343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 74/149] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:46:59 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=881cc16b876541819935dfa513a16e5d => generated 608 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:46:59.953487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed2cc4-68ad-4032-b414-8f70067e8d79 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.953977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed2cc4-68ad-4032-b414-8f70067e8d79 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.953977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5ed2cc4-68ad-4032-b414-8f70067e8d79 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.955729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 76/150] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:46:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:46:59.960915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:46:59.961479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:46:59.961965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:46:59.987795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:46:59.989233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:00.075467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 75/151] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:46:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:00.107735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ed2cc4-68ad-4032-b414-8f70067e8d79 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.238268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5ed2cc4-68ad-4032-b414-8f70067e8d79 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.284065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.284874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.285156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.285427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.298663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-77d476c0-f1ff-432f-8585-fd45880b0aa3 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:00.307819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:00.308834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:00.309610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:00.312976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 77/152] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/roles/reader => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:00.318949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.333673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:00.380335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602af78f-4f09-4fe8-94a6-b7b55fa6f063 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.380678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602af78f-4f09-4fe8-94a6-b7b55fa6f063 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.380986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-602af78f-4f09-4fe8-94a6-b7b55fa6f063 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.382426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 78/153] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:00 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:00.388736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.389020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.389299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.405967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:00.428363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:00.429560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:00.507108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 79/154] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:00 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:00.530892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1c0238-f7a1-4ac8-b087-bc88756a2568 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.532122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.593274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.594349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.594943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.595525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.596485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:00.596996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:00.598809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:00.616863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 76/155] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 298 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:00.623786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.640445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bde84f0-0ed9-48f7-99e2-5ea80995c337 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.676071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.676864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.677192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.677471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.679467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:00.691683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:00.701125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:00.702246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:00.703137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 80/156] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/roles/member => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:00.708856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.715415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:00.750739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179b0436-1dcf-4601-b32a-1d10e59ec090 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.790863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.792573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.793166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.793653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] PATH_INFO: `/v3/users/8a7efcb9024144239e6369dc3c16e746` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.795214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] RBAC: Authorizing `identity:get_user(user_id=8a7efcb9024144239e6369dc3c16e746)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:00.797592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:00.799498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 77/157] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/users/8a7efcb9024144239e6369dc3c16e746 => generated 289 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:00.805278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.821521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a541e19-651f-4e88-8b47-da41ceccc038 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.862015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.862748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.863051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.863323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.864077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:00.864382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:00.865971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:00.875869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 81/158] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:00.881445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.918050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51884de1-c068-4230-bb15-09e91e79b994 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:00.946529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:00.971057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:00.972294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.972889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.973373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.975476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:00.977348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c33dd9ef-2989-40a0-8609-36d59322241d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:00.980029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 78/159] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:00.987784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e13a6-bb9e-44db-9b83-f47bee8b24fe None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:00.988416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e13a6-bb9e-44db-9b83-f47bee8b24fe None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:00.988989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4e13a6-bb9e-44db-9b83-f47bee8b24fe None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:00.990685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 79/160] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:00 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:00.991687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e13a6-bb9e-44db-9b83-f47bee8b24fe None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:00.993984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c06aaa8-20ff-4f19-acb5-d09d436b71e7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.019209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.033780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.033780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.033780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.033780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.034177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.035492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179a1475-64ea-4efb-83ed-cbc0e270f813 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.042132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 82/161] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:00 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:01.049155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.049155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.049155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.062358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:01.063865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:01.134595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 83/162] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:01.136408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:01.188318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4e13a6-bb9e-44db-9b83-f47bee8b24fe None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.194109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.240759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f916548-d59a-468f-8305-3a90b6f4ed9b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.248448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.249704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.250250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.250723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/8a7efcb9024144239e6369dc3c16e746/roles/b65cce2fe3734269906540559fc82939` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.256180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, user_id=8a7efcb9024144239e6369dc3c16e746, role_id=b65cce2fe3734269906540559fc82939)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.258098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.279586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.280299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.280640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.280913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.289170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 80/163] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:00 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/users/8a7efcb9024144239e6369dc3c16e746/roles/b65cce2fe3734269906540559fc82939 => generated 0 bytes in 298 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:01.290261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:01.358203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.359861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.451153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.471203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.488906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 84/164] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 354 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:01.494585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:01.590172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d34a522-fe3f-4de4-b65a-eee5cede7401 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.613855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44b09952-2787-4762-b8c5-96a5206ad97c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.646533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.647391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.647679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.647959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.662607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.663206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:01.663503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.663783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.664107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] PATH_INFO: `/v3/projects/fe820d8d7de0479598637dfa23756659` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.671557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] RBAC: Authorizing `identity:get_project(project_id=fe820d8d7de0479598637dfa23756659)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.673055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.675330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 85/165] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/projects/fe820d8d7de0479598637dfa23756659 => generated 312 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:01.676615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:01.677467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:01.678522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 81/166] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 388 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:01.681191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:01.691818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:01.727154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.806342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e388171-0579-4660-acec-1731d521cde2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.843840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94542a80-4079-4a8a-bf08-e14380eb1036 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:01.844370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.845004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.845389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.845671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.846670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=fe820d8d7de0479598637dfa23756659, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:01.847013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.848666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d46ca4-da5d-41da-9f6b-60888602115c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.860792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 86/167] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=fe820d8d7de0479598637dfa23756659&role.id=881cc16b876541819935dfa513a16e5d => generated 608 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:01.865347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d203e37-1a11-484e-ad0a-3b6a40ff5f63 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.865596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d203e37-1a11-484e-ad0a-3b6a40ff5f63 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.865927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d203e37-1a11-484e-ad0a-3b6a40ff5f63 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.867570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 87/168] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:01.873991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.874298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.874658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.887757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:01.903029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:01.903720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:01.904013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:01.904296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:01.905064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:01.905380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:01.906788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:01.917340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:01.918886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:01.920525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 82/169] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:01.936131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.018944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 88/170] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:02.054273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.119779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:02.181026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a2b068d-d76a-492a-8fc8-1e84cc5ece8e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:02.236093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:02.237043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:02.237327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:02.237601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:02.252443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:02.256717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cbe3ca6-c5b7-44c3-8e6f-8cad92aa9541 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:02.261132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:02.261991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:02.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 89/171] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:02 2022] GET /identity/v3/roles/admin => generated 83 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:02.271684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.355513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:02.356423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:02.356715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:02.356996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] PATH_INFO: `/v3/users/801a75d4248e4a86a827af56143e1da2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:02.358281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] RBAC: Authorizing `identity:get_user(user_id=801a75d4248e4a86a827af56143e1da2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:02.359761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:02.361420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 83/172] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:01 2022] GET /identity/v3/users/801a75d4248e4a86a827af56143e1da2 => generated 275 bytes in 426 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:02.366808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:02.476172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4aaa560-4ea3-433b-babc-8b9ea7252b33 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:02.498418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:02.563337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:02.564030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:02.564316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:02.564603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:02.565381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:02.565687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:02.567131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:02.593380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 90/173] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:02 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 322 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:02.600006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.621863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe4bfe27-2bc9-4dc0-8b62-7d695f90af9d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:02.739178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-247a27f3-fde7-4597-a374-ebf93c7164c7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:02.743549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:02.744890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:02.745199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:02.745484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:02.783491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:02.784388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:02.784671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:02.784944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] PATH_INFO: `/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:02.788945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] RBAC: Authorizing `identity:get_user(user_id=f00ceffcf1cf4b95908b3794e7b9b31d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:02.790405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:02.792426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 91/174] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:02 2022] GET /identity/v3/users/f00ceffcf1cf4b95908b3794e7b9b31d => generated 247 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:02.799215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:02.831133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:02.904123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:02.905752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:03.006058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.043096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4531840-eb2d-47e6-b8c9-14d8902d9852 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:03.087323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 84/175] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 721 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:47:03.093136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.132531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:03.133143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.133628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.133928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.285227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:03.286643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:03.295581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.303917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b10e0d8e-6246-4b68-a02e-88aaed88992a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:03.371555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:03.372450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.372740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.373023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.374314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:03.375818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c5b882c-1f47-427f-8d9a-64771caf419b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:03.377552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 85/176] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 285 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:03.378742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9284f63-5d1d-403b-938d-cf11f7d2a3be None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.379061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9284f63-5d1d-403b-938d-cf11f7d2a3be None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.379343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9284f63-5d1d-403b-938d-cf11f7d2a3be None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.380593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 86/177] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:03 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:03.388059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8895911-de17-445c-99af-b055861255eb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.388356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8895911-de17-445c-99af-b055861255eb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.388697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8895911-de17-445c-99af-b055861255eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.399486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 92/178] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 601 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:03.400688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.415793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f8895911-de17-445c-99af-b055861255eb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:03.422840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f8895911-de17-445c-99af-b055861255eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:03.458322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.514220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1dc19d6-670c-44f6-a14a-90d77d0be5ef None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:03.525981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.553631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 87/179] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:03.554383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8895911-de17-445c-99af-b055861255eb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.602940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:03.605467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.607177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.607674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.608718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=801a75d4248e4a86a827af56143e1da2) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:03.610281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:03.612025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:03.624780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 93/180] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/role_assignments?user.id=801a75d4248e4a86a827af56143e1da2&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=496aed2e00104d4480f0efe375a79409 => generated 608 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:03.627063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.743140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8895911-de17-445c-99af-b055861255eb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:03.825387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a0acee1-f3dd-4de2-b9fb-b5451d583974 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:03.832571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:03.833960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.834450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.834982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.836511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:03.838293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:03.844609 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 88/181] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 291 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:03.850929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.880011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:03.881307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.881594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.881875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.884006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.898757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-847b71b0-ea7f-4d19-999c-86fd61d34988 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:03.909363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:03.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:03.911105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 94/182] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 284 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:03.913236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb7bbf-05a8-40da-9d72-a243eccd9c67 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:03.913236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb7bbf-05a8-40da-9d72-a243eccd9c67 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:03.913538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8efb7bbf-05a8-40da-9d72-a243eccd9c67 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:03.915164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 95/183] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:03 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:03.919943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb7bbf-05a8-40da-9d72-a243eccd9c67 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:03.965596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:03.983619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff537f16-a5e4-4d31-9816-dff4f1b492d4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.032708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.033383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.034049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.034327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.035386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=f00ceffcf1cf4b95908b3794e7b9b31d) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:04.035700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.037151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9a5bc85-d2ff-4730-942a-2ed63ed6c896 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.044171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 89/184] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/role_assignments?user.id=f00ceffcf1cf4b95908b3794e7b9b31d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:04.045442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410b68a-c13d-42f5-804a-3fdf9d701c08 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.045730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410b68a-c13d-42f5-804a-3fdf9d701c08 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.046014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2410b68a-c13d-42f5-804a-3fdf9d701c08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.057357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8efb7bbf-05a8-40da-9d72-a243eccd9c67 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.084003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2410b68a-c13d-42f5-804a-3fdf9d701c08 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:04.085154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2410b68a-c13d-42f5-804a-3fdf9d701c08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:04.106728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.108308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.108823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.109333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.110862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:04.111241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.114067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-973bbfdd-285c-41d1-ab73-a12441fad25c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.126095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 96/185] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:03 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:04.127657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813530d6-b036-43ed-97c4-505eaeaa8110 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.127958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813530d6-b036-43ed-97c4-505eaeaa8110 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.128244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813530d6-b036-43ed-97c4-505eaeaa8110 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.129753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 97/186] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:04.133146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813530d6-b036-43ed-97c4-505eaeaa8110 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.186564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 90/187] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:04.190070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.190554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.191038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.215424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:04.217178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:04.231247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.252412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.264880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813530d6-b036-43ed-97c4-505eaeaa8110 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.306248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.307762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.308055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.308692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.314916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 91/188] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:04.316091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.327393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.328950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.331250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 98/189] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:04.337233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.438079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2f9e9e3-cf85-4acd-aac5-b9d0fd280991 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.452133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b03c67f-956c-40aa-aed8-91f0ebc50e74 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.472349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.473141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.473419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.473690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.483834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:04.486134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.486817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.487110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.487391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:04.491946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:04.492856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:04.493659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 92/190] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:04.494415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.501188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.507712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.558408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.559956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.595145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74486d8c-345f-4282-ba79-b6fa0c4f85e0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.627612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.628400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.628681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.628954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.630120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.631688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7c5f74-470e-4f63-a26b-8b5811a54ec6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 93/191] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66860017-bcce-4c70-9004-1fa09e422959 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66860017-bcce-4c70-9004-1fa09e422959 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66860017-bcce-4c70-9004-1fa09e422959 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 94/192] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:04.638234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66860017-bcce-4c70-9004-1fa09e422959 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.658912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 99/193] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 322 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:04.659732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.708172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.709361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.740984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66860017-bcce-4c70-9004-1fa09e422959 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.757835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2332b23-f512-4c9f-b1ad-8a81f80d611d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.774650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.776000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.776291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.776566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.777309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:04.777616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.779012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2260d3c3-0a32-447b-86dd-0839d06d32b0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.786880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 95/194] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:04.788045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.788338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.788621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.794241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:04.795466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:04.795752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:04.796029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:04.796741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:04.798188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:04.804735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:04.805931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:04.818712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 100/195] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:04 2022] POST /identity/v3/users => generated 297 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:04.818712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.889123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 96/196] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:04.890223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:04.957404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e8154-3018-4a31-8901-0c128f6d477e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:04.967519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:04.972802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.007728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.008602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.008879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.011879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.011879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.012742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.014449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 101/197] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.016004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fcee9eb-e6f7-4021-97d5-bfea2ca51f18 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.016432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.057991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.058953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.059247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.059531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.061141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.062607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.064317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 97/198] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:04 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.065089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.139658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a01b813-dddc-48d8-85f2-2e6831507f35 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.195619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.196434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.196612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-560d96f9-89d9-4a01-bdd0-847ad9112223 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.196709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.196983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.211534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:05.222679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:05.223574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:05.224369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:05.224955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 102/199] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/roles/reader => generated 84 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:05.225491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.235753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.241474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.242403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.242891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.243237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.244189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=ec20963dea194be5a701098548c905e6) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:05.244574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.246474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.252960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 98/200] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/role_assignments?user.id=ec20963dea194be5a701098548c905e6&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.253891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.270033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.355099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94f1831e-73b3-47c2-9766-ba5115a39f01 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.391798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b634065-a172-47e4-a3ce-7f24862454a4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.398651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.399346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.399625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.399898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.450617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.451380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.452647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.452926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.453704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:05.454012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.455457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.475740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 99/201] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 222 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.485403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.520888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.522765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.587392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.616618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.665429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 103/202] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 440 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:05.667011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.701666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71698add-2937-4953-8ce9-154841e78ce0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.739848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.741187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.741762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.742231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] PATH_INFO: `/v3/users/8a7efcb9024144239e6369dc3c16e746` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.743703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] RBAC: Authorizing `identity:get_user(user_id=8a7efcb9024144239e6369dc3c16e746)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.746357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.748544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 100/203] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/users/8a7efcb9024144239e6369dc3c16e746 => generated 289 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.754183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.831858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e661c33f-0869-4ada-8ce6-2ff3bce1cc3c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.889189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:05.892462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:05.892743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:05.893013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:05.895760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:05.897225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:05.904429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 104/204] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:05.905270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:05.913081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:05.940175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8510e01-eede-4ed8-a2c8-9e11e1bdca27 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:05.989543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:06.007466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:06.008447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:06.008932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:06.009405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:06.060722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed8f386-2ae1-45ce-94da-8903e08b62d8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:06.098240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:06.099340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:06.099952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:06.100425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:06.101466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=ec20963dea194be5a701098548c905e6) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:06.101971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:06.104361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:06.111095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 105/205] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/role_assignments?user.id=ec20963dea194be5a701098548c905e6&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=496aed2e00104d4480f0efe375a79409 => generated 259 bytes in 206 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:06.203100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:06.204506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:06.334045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 101/206] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 580 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:06.340515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:06.427131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:06.479247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3c81f25-8936-4eee-96a7-19809d982580 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:06.521068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:06.521950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:06.522228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:06.522499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:06.523712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:06.525161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aff0f26f-6a73-4205-aca7-f50bf6d23f79 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:06.530875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 106/207] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:06 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:06.532683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:06.588876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:06.703716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e58cc644-cb25-4b52-8245-36e928078e3b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:06.765552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:06.766576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:06.767084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:06.767578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:06.768622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c, user.id=8a7efcb9024144239e6369dc3c16e746) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:06.769153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:06.770829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6a116a5-7e8f-476b-8ecb-3cc010406dfe None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:06.783486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 102/208] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:06 2022] GET /identity/v3/role_assignments?user.id=8a7efcb9024144239e6369dc3c16e746&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=b65cce2fe3734269906540559fc82939 => generated 608 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:07.152063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79863e-53f4-4e26-a681-060a24f68d5b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.152762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79863e-53f4-4e26-a681-060a24f68d5b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.153264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b79863e-53f4-4e26-a681-060a24f68d5b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.154951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 107/209] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:07.160228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e409e30-2ca7-46f8-aaae-361f46538186 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.161072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e409e30-2ca7-46f8-aaae-361f46538186 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.161545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e409e30-2ca7-46f8-aaae-361f46538186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.183791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3e409e30-2ca7-46f8-aaae-361f46538186 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:07.185312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3e409e30-2ca7-46f8-aaae-361f46538186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:07.266679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 103/210] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:07.294427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b79863e-53f4-4e26-a681-060a24f68d5b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.395095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74ed64-3da2-404f-a329-d2af3bcfe90f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.395095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74ed64-3da2-404f-a329-d2af3bcfe90f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.395095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a74ed64-3da2-404f-a329-d2af3bcfe90f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.396175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 104/211] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:07.403064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.403603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.404094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.417196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b79863e-53f4-4e26-a681-060a24f68d5b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:07.427112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:07.428560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:07.445006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:07.464183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:07.465228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.465589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.465875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.468019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:07.479806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:07.489931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:07.490828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:07.491617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:07.492201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 108/212] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:07 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:07.500270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.509068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 105/213] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:07.533690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.621408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a07c473-d74c-4f44-83fd-b160530b2028 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:07.655361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5ca824b-a5f5-45f3-a9c8-8df293eca998 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:07.659416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:07.660318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.660820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.661293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.662248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:07.662753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:07.664486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:07.676071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 109/214] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:07 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:07.682299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.691532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:07.692335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.692614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.692885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.694054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:07.695533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:07.697193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 106/215] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:07 2022] GET /identity/v3/domains/default => generated 199 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:07.702602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.721547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:07.753435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:07.804453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-732f9997-6d31-43a6-bf8e-08483f257106 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:07.805023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d32093f7-4de1-4c49-aa6e-979032661739 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:07.841357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:07.842124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.842416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.842690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.843655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:07.844932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:07.845194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:07.846240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.846719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.847218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] PATH_INFO: `/v3/users/35bca23fde5b4db3ba47015869e1591c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.865655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] RBAC: Authorizing `identity:get_user(user_id=35bca23fde5b4db3ba47015869e1591c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:07.867691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985a0186-30fc-43e0-9989-e122ca6fa451 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:07.867985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 107/216] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:07 2022] POST /identity/v3/users => generated 297 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:47:07.874554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 110/217] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:07 2022] GET /identity/v3/users/35bca23fde5b4db3ba47015869e1591c => generated 297 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:07.876907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:07.878699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e13644-9260-43f1-9d89-c077f23f7640 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.879303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e13644-9260-43f1-9d89-c077f23f7640 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.880026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30e13644-9260-43f1-9d89-c077f23f7640 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.881953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 111/218] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:07 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:07.888459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:07.888753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:07.889010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:07.907475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:07.909134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:07.972467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.007076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5693abb-0340-4859-ad39-4ce45166b539 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.012001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 112/219] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:08.039453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.049506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.050295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.050681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.051084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.093595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.145129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.147115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.166731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0bc3d35-3905-4143-9c8d-89790b58d837 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.216553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.217471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.217849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.218192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.232642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:08.240972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:08.241885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:08.242685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:08.243351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.246034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 113/220] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:08.252570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.270180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 108/221] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:07 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 394 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:08.276015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.378847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63e6b1a-2e3e-466d-9231-5c573bef603c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.381605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.387554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d50899f0-6b70-4939-8fe9-3698b9576965 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.418497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.419829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.420326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.420794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.421955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:08.422549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.424149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.424527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.425019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.425299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.425571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.426896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.428457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.430188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 109/222] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:08.436279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 114/223] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:08.440773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.441925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.484782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.545870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb76502-8830-4bd7-a9cf-5989fe95c553 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.548425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f31f029f-90f4-49ec-8fc2-d102db6dbdb5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.581854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.582639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.583007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.583379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.584265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=35bca23fde5b4db3ba47015869e1591c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:08.584525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.585791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.586066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e31647-17b7-44e9-8ab1-a99725bf2a75 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.587152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.587626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.588326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.590258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.591981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.592330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 110/224] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/role_assignments?user.id=35bca23fde5b4db3ba47015869e1591c&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=881cc16b876541819935dfa513a16e5d => generated 259 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:08.594271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 115/225] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:08.598666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3649360-2a6d-4db4-9a8f-7f05467b0f33 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.598666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3649360-2a6d-4db4-9a8f-7f05467b0f33 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.598666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3649360-2a6d-4db4-9a8f-7f05467b0f33 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.604773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.605357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 111/226] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:08 2022] GET /identity => generated 270 bytes in 12 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:08.608943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.609216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.609478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.647448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.651019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:08.652518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:08.782870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 112/227] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:08.783033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa2e1670-60e6-4673-b01c-5f93684f04a4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:08.802508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.833131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:08.834360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:08.834877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:08.836379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:08.838496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:08.841429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:08.844118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 116/228] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:08.851683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:08.876139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:08.993465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c9b5141-6893-4b0c-bace-d6e686a10b2b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.044479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbfe8af3-8574-4a0a-b4fd-1a08e20bf869 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.046111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:09.082342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.083275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.083593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.083869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/ec20963dea194be5a701098548c905e6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.085861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, user_id=ec20963dea194be5a701098548c905e6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:09.087344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:09.088640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.089660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.089955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.090232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.104510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 117/229] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:08 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/ec20963dea194be5a701098548c905e6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:09.129331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:09.139113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:09.140080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:09.140875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 113/230] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:08 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 338 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:09.146639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:09.307209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc5920aa-3273-4bd4-8b3f-41522464d553 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.344607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.345333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.345620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.345896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.346649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:09.346982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:09.348363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:09.357665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 118/231] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:09 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:09.363558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:09.476328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:09.484142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3e356-dcc0-404a-a425-a7f5364c5e52 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.539149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.540324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.540808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.541287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.547483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:09.549120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:09.550984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 114/232] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:09 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:09.557112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:09.589901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:09.688367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4549e9-d064-4a86-8290-71c2819fa8de None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.744978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.746160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.746616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.747099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.748548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:09.750231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef5da25d-3ed2-4f26-8fe0-468d24817eef None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:09.753221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 119/233] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:09 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:09.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:09.886061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2323d27-ba6f-49c4-ad37-1969fc7d4876 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:09.920324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:09.921524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:09.921980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:09.922426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/ec20963dea194be5a701098548c905e6/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:09.924888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, user_id=ec20963dea194be5a701098548c905e6, role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:09.926521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7adbe1-de6c-413b-923e-1f681c77622e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:09.943144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 115/234] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:09 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/ec20963dea194be5a701098548c905e6/roles/496aed2e00104d4480f0efe375a79409 => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:10.574229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8abfcc6-1e59-4aa0-b260-685eda61bba3 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:10.575288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8abfcc6-1e59-4aa0-b260-685eda61bba3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:10.575787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8abfcc6-1e59-4aa0-b260-685eda61bba3 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:10.577659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 120/235] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:10 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:10.582655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78bc12-e848-4c4d-afdd-7126a5738ba3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:10.583519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78bc12-e848-4c4d-afdd-7126a5738ba3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:10.583992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d78bc12-e848-4c4d-afdd-7126a5738ba3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:10.601807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0d78bc12-e848-4c4d-afdd-7126a5738ba3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:10.603392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0d78bc12-e848-4c4d-afdd-7126a5738ba3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:10.622517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:10.678882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 116/236] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:10.715045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8abfcc6-1e59-4aa0-b260-685eda61bba3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:10.828776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2c276-9a93-4159-9913-33de129edd3c None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:10.829331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:10.829459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2c276-9a93-4159-9913-33de129edd3c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:10.829680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dad2c276-9a93-4159-9913-33de129edd3c None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:10.831281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 117/237] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:10.836446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:10.836732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:10.837008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:10.847202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8abfcc6-1e59-4aa0-b260-685eda61bba3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:10.863321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:10.864520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:10.914982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:10.915820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:10.916197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:10.916475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:10.940572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:10.953796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:10.954661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:10.955481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 121/238] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:10 2022] GET /identity/v3/roles/reader => generated 84 bytes in 241 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:10.963834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:10.992068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 118/239] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:10 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:11.035879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.150848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46abade2-c4e5-4bf9-8d49-4edc611006eb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.190511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.208945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43adde10-3107-4103-ba29-79c1fd246bbf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.224371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.225053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.225334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.225606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.226357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:11.226659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.228089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.242612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 122/240] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:10 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 280 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:11.248564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.254532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.255421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.255704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.255977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.261068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.262703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.264565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 119/241] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/domains/default => generated 199 bytes in 229 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:11.270591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.276194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.369536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3150ecf7-4a27-4d86-bd5a-a68282661b17 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.406394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e5d4b3-65ae-4d3b-9d56-2a7bb64b93a0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.413075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.413960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.414239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.414511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] PATH_INFO: `/v3/users/5b108bcedf7d42b7814ea7369c7c927d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.434604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] RBAC: Authorizing `identity:get_user(user_id=5b108bcedf7d42b7814ea7369c7c927d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.437082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05168e53-0ea6-42cf-8560-0f14ddf9f7b7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.439687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 123/242] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/users/5b108bcedf7d42b7814ea7369c7c927d => generated 297 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:11.441657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18986f4f-bc51-4051-a69c-6f94bbfe8487 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.441941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18986f4f-bc51-4051-a69c-6f94bbfe8487 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.442212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18986f4f-bc51-4051-a69c-6f94bbfe8487 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.444371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 124/243] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:11.445234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18986f4f-bc51-4051-a69c-6f94bbfe8487 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.453840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.454826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.455306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.455781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.456713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.458355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7472724-f570-470a-b316-9b2143744af3 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.481016 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 120/244] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:11 2022] POST /identity/v3/users => generated 293 bytes in 211 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:11.483005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.483527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.484151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.505201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:11.506972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:11.509622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.534198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.572263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18986f4f-bc51-4051-a69c-6f94bbfe8487 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.591764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 121/245] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:11.612251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.612999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.613373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.613721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.635447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.690452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.691979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-725a865b-5778-48d9-9d95-5d0cc4f3b625 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.769575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.776053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32556d97-3a5a-4594-ab4b-db05360ae4ef None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.804835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 125/246] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 360 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:11.807121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3887c894-54cf-4669-b00a-5413944bef89 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.807421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3887c894-54cf-4669-b00a-5413944bef89 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.807765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3887c894-54cf-4669-b00a-5413944bef89 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.809280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 126/247] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:11 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:11.811867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3887c894-54cf-4669-b00a-5413944bef89 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.822896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.823705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.824048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.824331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.840448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:11.843546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-879da078-ff3e-4b1f-aa4a-187d82642a5b None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:11.851719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:11.852607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:11.853974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 122/248] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/roles/member => generated 84 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:11.855082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.855082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.855082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.879908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:11.881395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:11.939049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3887c894-54cf-4669-b00a-5413944bef89 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:11.974257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 123/249] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:11 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 121 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:11.974969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:11.984340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:11.985278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:11.985616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:11.985953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:11.987763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:11.989909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:11.992370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 127/250] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.000786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.087237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:12.091000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9bc1ef1-2eaf-415d-926d-d5d0f0362bbd None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.123280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:12.132647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c484eda8-cfcb-4210-a356-2b4a6f39da8e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.134509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.135690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.136004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.136291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.137166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:12.137353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.138754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.147874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 124/251] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.148812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.186808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.188054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.188768 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.189700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.191021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=5b108bcedf7d42b7814ea7369c7c927d) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:12.191770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.194182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.201873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 128/252] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:11 2022] GET /identity/v3/role_assignments?user.id=5b108bcedf7d42b7814ea7369c7c927d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=b65cce2fe3734269906540559fc82939 => generated 259 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:12.207243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.393801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-865042ff-e82c-4dc0-9573-3d11055acb6f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.458109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b8264ec-f2f5-4c87-91a5-b434cb51b5b1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.465067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:12.471895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.472744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.473031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.473304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.496374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:12.504236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:12.505083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:12.505878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 125/253] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:12 2022] GET /identity/v3/roles/admin => generated 83 bytes in 357 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:12.515314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.539178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.540076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.540358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.540631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] PATH_INFO: `/v3/users/35bca23fde5b4db3ba47015869e1591c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.543706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] RBAC: Authorizing `identity:get_user(user_id=35bca23fde5b4db3ba47015869e1591c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.545160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.546971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 129/254] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:12 2022] GET /identity/v3/users/35bca23fde5b4db3ba47015869e1591c => generated 297 bytes in 344 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.553917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.620818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:12.690249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0d5e8c-aa00-4780-b47f-01146412926b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.726065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.726746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.727200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.727484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.728781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:12.729165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.731017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-141b9f4b-55f4-4bd6-98ab-99f7e8e79447 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.739672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 126/255] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:12 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.740916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f13edbb-9dd0-4fb6-95c1-ff1c622a39f4 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.741315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f13edbb-9dd0-4fb6-95c1-ff1c622a39f4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.741593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f13edbb-9dd0-4fb6-95c1-ff1c622a39f4 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.747142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 127/256] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:12 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:12.747956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f13edbb-9dd0-4fb6-95c1-ff1c622a39f4 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.777012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6ce0a8-ec23-4439-ac2c-9abb86ade31d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.857757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f13edbb-9dd0-4fb6-95c1-ff1c622a39f4 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:12.863022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.863892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.864172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.864444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.870750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.872241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-447b2601-747c-4aa3-af99-35975c087f1b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.873968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 130/257] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:12 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 320 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.879475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.879796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.880076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.886545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:12.917610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:12.918828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:12.924331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:12.925212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:12.925488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:12.925758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:12.927046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:12.929483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:12.937040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 128/258] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:12 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:12.937859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:12.976494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:13.016814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 131/259] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:12 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 142 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:13.017712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:13.165821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58db3866-5b42-4655-9424-6ee95b123c49 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:13.179721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657453e8-9dcd-4cf0-8519-e368e1764758 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:13.230332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:13.230966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:13.231250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:13.231549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:13.261421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:13.262348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:13.262629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:13.262929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/35bca23fde5b4db3ba47015869e1591c/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:13.267183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, user_id=35bca23fde5b4db3ba47015869e1591c, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:13.268654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:13.294571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 129/260] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:12 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/35bca23fde5b4db3ba47015869e1591c/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 357 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:13.295975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:13.297943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:13.361177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:13.362666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:13.386168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:13.515152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dce95da5-dfab-433a-8fde-7df5415bdea2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:13.579120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:13.579965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:13.580255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:13.580535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:13.601979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 132/261] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 584 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:13.604484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:13.608327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:13.617116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:13.618101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:13.619900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 130/262] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 323 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:13.626504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:13.739715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:13.775838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:13.781205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-323c20e1-2eea-4e83-ac5c-08aed9dcaea4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:13.787372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f88b83f-b958-42c3-9358-baee8759c0e2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:13.855876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:13.856802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:13.857076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:13.857346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:13.857441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:13.858191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:13.858482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:13.858783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:13.859548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:13.859859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:13.861263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:13.863534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:13.865064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:13.867114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 133/263] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 259 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:13.873656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:13.879131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 131/264] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 253 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:13.885188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.032975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80980908-d3d3-4470-8046-7b9aac53a489 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.057548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70dd6902-be68-48a3-93ee-01871d0520a5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.076017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.076908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.077188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.077461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] PATH_INFO: `/v3/users/ec20963dea194be5a701098548c905e6` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.083489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] RBAC: Authorizing `identity:get_user(user_id=ec20963dea194be5a701098548c905e6)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.084948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.086840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 132/265] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/users/ec20963dea194be5a701098548c905e6 => generated 283 bytes in 202 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:14.092583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.104753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.105429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.105716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.105994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.106865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=ec20963dea194be5a701098548c905e6) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:14.107183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.108653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8820c185-ad02-475a-a384-e451975a9b95 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.129511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 134/266] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:13 2022] GET /identity/v3/role_assignments?user.id=ec20963dea194be5a701098548c905e6&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 256 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:14.145021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:14.252246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5391695e-8dea-44ca-8d94-073ad41c62d8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.297495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.298079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.298366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.298645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.315035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ab113-40dc-4578-b01b-f9fc99d92431 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.315336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ab113-40dc-4578-b01b-f9fc99d92431 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.315603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f8ab113-40dc-4578-b01b-f9fc99d92431 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.317198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 135/267] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:14.322994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.323575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.324048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.334807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:14.371437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:14.372669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:14.408130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.409943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.475799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:14.526795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 136/268] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 199 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:14.583071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 133/269] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:14 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 491 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:47:14.590206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.595640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.720320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b2bf57-1a14-4bfc-ab16-9fbcd83c1c46 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.732928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86659ae5-8639-4740-99f2-edb567380f09 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.751638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:14.758060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.759341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.760164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.760441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.761809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.763595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f0da866-002a-4361-ba65-b09dba46bc4c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.769358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 134/270] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:14 2022] GET /identity/v3/domains/default => generated 199 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:14.769358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f21e6ab-1b2e-4c01-8db4-5491f03df21e None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.769358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f21e6ab-1b2e-4c01-8db4-5491f03df21e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.769358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f21e6ab-1b2e-4c01-8db4-5491f03df21e None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.769358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 135/271] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:14.774446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f21e6ab-1b2e-4c01-8db4-5491f03df21e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.777871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:14.785404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.786932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.787447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.788353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.789892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.791851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-718fa316-15f2-4fc5-8f89-e21f65451bfa None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.793774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 137/272] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:14 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 204 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:14.800074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.800694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.801270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.827836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:14.829352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:14.898716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f21e6ab-1b2e-4c01-8db4-5491f03df21e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:14.903078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 138/273] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:14 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:14.904629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:14.960168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:14.961204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:14.961710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:14.962288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:14.963285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:14.965061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:14.998978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 136/274] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:14 2022] POST /identity/v3/users => generated 293 bytes in 217 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:15.000373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.029209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64685eff-e3c1-4e1b-b52c-a2247f78c667 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.062245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.076688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.077597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.077735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.077944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.078840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=ec20963dea194be5a701098548c905e6) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:15.079352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.080729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e06bf9c0-faa3-46c5-bbc9-75ebd9cefb9e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.087459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 139/275] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:14 2022] GET /identity/v3/role_assignments?user.id=ec20963dea194be5a701098548c905e6&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=496aed2e00104d4480f0efe375a79409 => generated 608 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:15.089187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e0c2d-37fa-47c3-a99f-6066acb65e7b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.089479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e0c2d-37fa-47c3-a99f-6066acb65e7b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.089758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f7e0c2d-37fa-47c3-a99f-6066acb65e7b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.091301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 140/276] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:15.097050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.097489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.097837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.099723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.119640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:15.121417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:15.170871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-471ba8ac-fdb6-463c-9386-6dff7362fe7d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.217378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.218738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.219326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.219802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.235152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:15.244854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:15.245854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:15.246807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:15.249993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 137/277] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:14 2022] GET /identity/v3/roles/member => generated 84 bytes in 250 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:15.255459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.263128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 141/278] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:15 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:15.288675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.389298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eeb6258-7ac1-4cf0-b279-7e28ef0f04cd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.405881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.433478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7530178b-273b-470d-896a-5f03d1e857bd None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.437153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.439037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.439370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.439651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.440421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:15.440729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.442115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.453434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 138/279] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:15.459176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.471250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.489073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.489883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.490173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.490448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.501958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:15.511021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:15.511954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:15.512747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 142/280] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/roles/reader => generated 84 bytes in 223 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:15.517176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.586187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6728746-be50-4211-9652-dc6da662cc18 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.627480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.628364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.628650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.628929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] PATH_INFO: `/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.641888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1d778c3-26b8-4ca1-9ccb-606dd704ab21 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.644415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] RBAC: Authorizing `identity:get_user(user_id=435e89328cab4cfc9f8cf5733b7a2c0d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.645907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.647833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 139/281] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d => generated 293 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:15.654226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.680012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.680747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.681033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.681363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.682157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] RBAC: Adding query filter params (name=reader) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:15.682524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.684037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.692552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.693631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 143/282] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:15.700125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.750702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:15.769657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ea16f71-e777-4c6e-9fef-8ccb4099cccf None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.808331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6b0e2e-9c03-4a54-8c15-4e5f124319fa None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:15.810889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.811563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.811849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.812126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.812957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, user.id=435e89328cab4cfc9f8cf5733b7a2c0d, scope.system=all) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:15.813271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.814728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.820440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 140/283] 10.222.0.18 () {56 vars in 1308 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/role_assignments?user.id=435e89328cab4cfc9f8cf5733b7a2c0d&scope.system=all&role.id=881cc16b876541819935dfa513a16e5d => generated 226 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:15.848574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:15.849490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:15.849776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:15.850054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] PATH_INFO: `/v3/users/5b108bcedf7d42b7814ea7369c7c927d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:15.851534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] RBAC: Authorizing `identity:get_user(user_id=5b108bcedf7d42b7814ea7369c7c927d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:15.853123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:15.855380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 144/284] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/users/5b108bcedf7d42b7814ea7369c7c927d => generated 297 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:15.861173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:15.997988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:16.002587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4812e7ca-c22b-4f35-9214-8dad376ed59a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:16.052078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:16.053275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.053748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.054202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.055628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:16.057443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-376fcc16-60bf-4361-b309-0167f9ef0c49 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:16.059469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 141/285] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:15 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:16.065677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:16.147307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d6041-c1df-4154-84cd-a40d840cfb85 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.147886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d6041-c1df-4154-84cd-a40d840cfb85 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.148350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f07d6041-c1df-4154-84cd-a40d840cfb85 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.149941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 142/286] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:16.155092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.155592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.156048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.180381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:16.182671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:16.223594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8004a4f-bcbc-4694-babf-5bbdfe88a6e9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:16.245603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:16.288598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:16.289779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.290274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.290847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/5b108bcedf7d42b7814ea7369c7c927d/roles/b65cce2fe3734269906540559fc82939` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.294621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, user_id=5b108bcedf7d42b7814ea7369c7c927d, role_id=b65cce2fe3734269906540559fc82939)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:16.296331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:16.313950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 143/287] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:16 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 160 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:16.319637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 145/288] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:16 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/users/5b108bcedf7d42b7814ea7369c7c927d/roles/b65cce2fe3734269906540559fc82939 => generated 0 bytes in 254 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:16.359015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:16.485782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:16.491821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614a764b-ca1a-407d-aa26-22673443d98a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:16.533100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:16.533921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.534261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.534543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.546457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:16.554555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:16.555524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:16.556962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 144/289] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:16 2022] GET /identity/v3/roles/member => generated 84 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:16.564509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:16.700383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b949aff9-c40b-4aef-bd58-2a9256fd89f3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:16.737309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:16.738350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.738859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.739329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.740361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:16.740866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:16.742535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:16.753091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 146/290] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:16 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:16.758909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:16.889453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5dd85ef-6b14-4127-99da-13baf810b31e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:16.930952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:16.932809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:16.933288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:16.933756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] PATH_INFO: `/v3/users/35bca23fde5b4db3ba47015869e1591c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:16.939501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] RBAC: Authorizing `identity:get_user(user_id=35bca23fde5b4db3ba47015869e1591c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:16.941255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:16.943274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 145/291] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:16 2022] GET /identity/v3/users/35bca23fde5b4db3ba47015869e1591c => generated 297 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:16.949503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:16.995364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:17.066203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a8c3321-17e3-437a-a6ba-f5aa1ffdcd99 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:17.111904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:17.112967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:17.113444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:17.113926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:17.205392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:17.206933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:17.259130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:17.320900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 147/292] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:16 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:17.326376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:17.374647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:17.486375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16ad0384-2d68-4226-a44d-158a44228547 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:17.529924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:17.530837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:17.531131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:17.531411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:17.532636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:17.534103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-961532bf-f84c-4bbf-8471-f43b4f6bdcac None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:17.536177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 146/293] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:17 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:17.542462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:17.686452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a5223df-129a-44e4-a70a-63a520c254f6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:17.723610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070068bc-b1b6-4d6c-9f35-ccf49f64c12b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:17.723906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070068bc-b1b6-4d6c-9f35-ccf49f64c12b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:17.724200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:17.724659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-070068bc-b1b6-4d6c-9f35-ccf49f64c12b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:17.724869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:17.725161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:17.725442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:17.726277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=35bca23fde5b4db3ba47015869e1591c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:17.726592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:17.727539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 147/294] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:17 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:17.730953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:17.734510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:17.735067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:17.735563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:17.741792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:17.742022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 148/295] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:17 2022] GET /identity/v3/role_assignments?user.id=35bca23fde5b4db3ba47015869e1591c&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=881cc16b876541819935dfa513a16e5d => generated 608 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:17.754448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:17.755716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:17.830042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 148/296] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:17 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:17.872128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:17.992056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90e6a889-3924-4362-b3e1-ba40054e18b9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.030795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.031620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.031905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.032183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.049703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:18.057913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:18.058883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:18.059674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 149/297] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:17 2022] GET /identity/v3/roles/reader => generated 84 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:18.066974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.093653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.174709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22585d17-94e1-4dcf-a789-bbfe89180695 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.214652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.215378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.215667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.215945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.216701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:18.217011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:18.218408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94588f5f-fb31-47e9-b572-3a034f0bd43a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:18.229506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 149/298] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:18.235084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.304222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.317772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe2d026-b7fa-45e8-8809-aa966b7b9977 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.318087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe2d026-b7fa-45e8-8809-aa966b7b9977 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.318345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe2d026-b7fa-45e8-8809-aa966b7b9977 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.319759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 150/299] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:18.326539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.327044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.327611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.350548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:18.352568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:18.364925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa456e6b-c2fc-4304-9d53-5b76dbbfd74c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.410081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.411403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.411894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.412369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] PATH_INFO: `/v3/users/c5c0af3abdd245cdae0e726530bd1937` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.430318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] RBAC: Authorizing `identity:get_user(user_id=c5c0af3abdd245cdae0e726530bd1937)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:18.433335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:18.436059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 150/300] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/users/c5c0af3abdd245cdae0e726530bd1937 => generated 293 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:18.442559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.450655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 151/301] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:18.473630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.488616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.558007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c37a9d06-48de-4982-b2a6-4852e634076c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.578613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.582452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf7d4b51-3b68-4b1e-a75f-a8d73cd24069 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.597189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.597587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.597869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.598144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.599003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, user.id=c5c0af3abdd245cdae0e726530bd1937, scope.system=all) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:18.599318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:18.600778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14a773e6-b015-402b-80f6-07f839dedd89 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:18.606566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 151/302] 10.222.0.18 () {56 vars in 1308 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/role_assignments?user.id=c5c0af3abdd245cdae0e726530bd1937&scope.system=all&role.id=b65cce2fe3734269906540559fc82939 => generated 226 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:18.607660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb730ca-6d16-421b-878f-a9c9284b768b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.607947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb730ca-6d16-421b-878f-a9c9284b768b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.608226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb730ca-6d16-421b-878f-a9c9284b768b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.609632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 152/303] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:18.615246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.615544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.615815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.627127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.629105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.629679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.630190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.632098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:18.633993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:18.635494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:18.637016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:18.637952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 152/304] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/domains/default => generated 199 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:18.643859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.736139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 153/305] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:18 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:18.751003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.762027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:18.773550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64c2fa00-8f65-4a4b-8c3b-361025c7dac7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.813561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.814554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.815086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.815579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] PATH_INFO: `/v3/groups` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.816466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:18.818087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:18.838980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 153/306] 10.222.0.18 () {60 vars in 1134 bytes} [Sun Jul 24 00:47:18 2022] POST /identity/v3/groups => generated 229 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:18.898368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55f73e83-7fbd-4f2c-a027-357ef54234c9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:18.915759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:18.940772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:18.941952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:18.942499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:18.943003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:18.956190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-359cb2a9-d2a4-4617-8c6a-ed829125a04a None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:18.965494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:18.966473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:18.967323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:18.970190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 154/307] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:18.976284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.055787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae71b8a4-bbf4-4705-8c03-47a118997b3a None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.056360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae71b8a4-bbf4-4705-8c03-47a118997b3a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.056747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae71b8a4-bbf4-4705-8c03-47a118997b3a None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.058272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 155/308] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:19.065852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.066120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.066328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.084632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:19.085815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:19.093142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aae62e66-f9e5-4401-962d-cedcffa8a1d8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.134202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.135887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.136362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.136809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.137728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:19.138204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.152341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 154/309] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:18 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:19.158424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.163080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:19.168404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 156/310] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:19 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:19.193373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.286756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-438ba8f5-441a-46f0-83ea-6c80e839a39d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.302679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e69918d-89a3-4ab6-b792-f14f96c72cab None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.304340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:19.339119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.339934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.340223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.340506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] PATH_INFO: `/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.341733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] RBAC: Authorizing `identity:get_user(user_id=435e89328cab4cfc9f8cf5733b7a2c0d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.341838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.342628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.342949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.343225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.343326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.345382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 155/311] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:19 2022] GET /identity/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d => generated 293 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:19.350747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.363416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:19.373096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:19.373974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:19.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:19.377532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 157/312] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:19 2022] GET /identity/v3/roles/reader => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:19.384053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.439139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:19.481180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e74a51a3-ba41-426f-9397-e84e2b6364a8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.517575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.518444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.518734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.519123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] PATH_INFO: `/v3/system/users/435e89328cab4cfc9f8cf5733b7a2c0d/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.520549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=435e89328cab4cfc9f8cf5733b7a2c0d, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.521988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.523651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b12598-1f66-4b62-878a-286dbd0a91b5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.532949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 156/313] 10.222.0.18 () {58 vars in 1334 bytes} [Sun Jul 24 00:47:19 2022] PUT /identity/v3/system/users/435e89328cab4cfc9f8cf5733b7a2c0d/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:19.563502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.564567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.565044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.565511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.566508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] RBAC: Adding query filter params (name=reader) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:19.567047 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.568675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.579999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 158/314] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:19 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:19.588851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.704298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7a9b18f-a30c-4e73-858b-b350a88e3f11 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.739991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.740876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.741165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.741444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] PATH_INFO: `/v3/users/5b108bcedf7d42b7814ea7369c7c927d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.742653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] RBAC: Authorizing `identity:get_user(user_id=5b108bcedf7d42b7814ea7369c7c927d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.744216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.745869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 157/315] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:19 2022] GET /identity/v3/users/5b108bcedf7d42b7814ea7369c7c927d => generated 297 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:19.751444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:19.782980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:19.863380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e4a4d6f-434a-4880-b35c-f7d95065483a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:19.899534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:19.900572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:19.901048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:19.901515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:19.972367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:19.974155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:19.999937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:20.083779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 159/316] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:19 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 333 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:20.094818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:20.099088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:20.202654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ae66d3-c5e6-46a0-b628-477ebb78ad70 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:20.237897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:20.239122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:20.239604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:20.240091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:20.241488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:20.243136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aad36da-06bf-4ba2-92c5-9367ae6adf33 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:20.245468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 158/317] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:20 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:20.251157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:20.359597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acf5679c-c11f-46fd-9dd7-adc91efaa296 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:20.384476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:20.402183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, scope.project.id=7d8f3d187861468eb86fbc334ae08015, user.id=5b108bcedf7d42b7814ea7369c7c927d) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:20.406799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37888720-eb8d-4953-97fd-287522e61ca9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:20.413793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 160/318] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:20 2022] GET /identity/v3/role_assignments?user.id=5b108bcedf7d42b7814ea7369c7c927d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=b65cce2fe3734269906540559fc82939 => generated 608 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:20.726999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ef936-4f64-4882-834c-d79959613dfc None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:20.727370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ef936-4f64-4882-834c-d79959613dfc None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:20.727557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220ef936-4f64-4882-834c-d79959613dfc None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:20.729430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 159/319] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:20.734372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c34d5-8692-49db-9e2c-780f27ed50fc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:20.734922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c34d5-8692-49db-9e2c-780f27ed50fc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:20.735280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-319c34d5-8692-49db-9e2c-780f27ed50fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:20.753255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-319c34d5-8692-49db-9e2c-780f27ed50fc None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:20.754494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-319c34d5-8692-49db-9e2c-780f27ed50fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:20.827467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 161/320] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:20 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:20.851195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220ef936-4f64-4882-834c-d79959613dfc None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:20.925004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:20.958852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220ef936-4f64-4882-834c-d79959613dfc None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:20.992215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:20.992936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:20.993222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:20.993497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.000738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc794cd-d80a-4534-a1c7-7c67149e7c90 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.001031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc794cd-d80a-4534-a1c7-7c67149e7c90 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.001320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cc794cd-d80a-4534-a1c7-7c67149e7c90 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.002694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 162/321] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:21.004332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:21.007772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.008068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.008350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:21.012739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:21.013603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:21.014418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:21.018919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 160/322] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:20 2022] GET /identity/v3/roles/reader => generated 84 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:21.019754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.026529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:21.027768 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:21.098550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 163/323] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:21.123432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.132586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e724b041-0d48-493a-a484-b36afe7cdd18 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.165213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:21.172961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.173616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:21.175110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.175402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.175720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.176548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:21.176869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:21.178997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:21.193981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 161/324] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:21 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:21.194994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.230397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01b7ce43-a348-4da1-8e1a-04116a8f9b2c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.265507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.266232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.266516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.266811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.267982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:21.269434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:21.271120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 164/325] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:21 2022] GET /identity/v3/domains/default => generated 199 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:21.277235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.304147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cccfe4d1-4927-4963-96b3-b890d3c37181 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.340271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.341154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.341447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.341720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] PATH_INFO: `/v3/users/c5c0af3abdd245cdae0e726530bd1937` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.342954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] RBAC: Authorizing `identity:get_user(user_id=c5c0af3abdd245cdae0e726530bd1937)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:21.344410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:21.346231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 162/326] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:21 2022] GET /identity/v3/users/c5c0af3abdd245cdae0e726530bd1937 => generated 293 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:21.351625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.382665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdbab17-e26b-4ae5-8348-6f1cc7e78d52 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.403911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:21.426939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:21.428660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.429428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.429820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.430074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] PATH_INFO: `/v3/groups` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.430841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] RBAC: Authorizing `identity:create_group()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:21.433122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebb1132a-aa04-42cd-9546-a3c8ee6c42d7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:21.462915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 165/327] 10.222.0.18 () {60 vars in 1134 bytes} [Sun Jul 24 00:47:21 2022] POST /identity/v3/groups => generated 226 bytes in 178 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:47:21.469971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13481f2-86a1-48cc-bc2d-f4301f8c19b9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.507462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.508383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.508667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.508946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] PATH_INFO: `/v3/system/users/c5c0af3abdd245cdae0e726530bd1937/roles/b65cce2fe3734269906540559fc82939` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.511129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] RBAC: Authorizing `identity:create_system_grant_for_user(user_id=c5c0af3abdd245cdae0e726530bd1937, role_id=b65cce2fe3734269906540559fc82939)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:21.512557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da59a7a3-be6c-4e57-a59c-764d39228f5d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:21.520349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 163/328] 10.222.0.18 () {58 vars in 1334 bytes} [Sun Jul 24 00:47:21 2022] PUT /identity/v3/system/users/c5c0af3abdd245cdae0e726530bd1937/roles/b65cce2fe3734269906540559fc82939 => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:47:21.686809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000a89a-4c04-4cd3-9957-cfd80046506d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.687184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000a89a-4c04-4cd3-9957-cfd80046506d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.687359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0000a89a-4c04-4cd3-9957-cfd80046506d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.688717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 166/329] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:21 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:21.694038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.694325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.694606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.713523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:21.714731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:21.785684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 164/330] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:21 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:21.809792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0000a89a-4c04-4cd3-9957-cfd80046506d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:21.925731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0000a89a-4c04-4cd3-9957-cfd80046506d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:21.963340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:21.964152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:21.964434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:21.964710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:21.981285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:21.989400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:21.990279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:21.991242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 167/331] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:21 2022] GET /identity/v3/roles/member => generated 84 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:21.999424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:22.068504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:22.120955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ae13bd7-b08e-4d57-91d4-e90ef0ab193e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:22.161281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:22.162003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:22.162240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:22.162520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:22.163315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:22.163628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:22.165000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:22.175058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 165/332] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:21 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:22.180332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:22.227263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:22.311944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b507df9-140a-4996-94fc-d30ef92a375e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:22.359433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:22.360256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:22.360539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:22.360869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] PATH_INFO: `/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:22.362053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] RBAC: Authorizing `identity:get_user(user_id=435e89328cab4cfc9f8cf5733b7a2c0d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:22.363590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fa93991-f069-4811-a857-5e7deb3e41e5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:22.365541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 168/333] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:22 2022] GET /identity/v3/users/435e89328cab4cfc9f8cf5733b7a2c0d => generated 293 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:22.372149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:22.489777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f72fca-ef86-409b-8033-e61c4039ef12 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:22.508460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:22.528450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:22.529110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:22.529398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:22.529681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:22.530511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] RBAC: Adding query filter params (role.id=881cc16b876541819935dfa513a16e5d, user.id=435e89328cab4cfc9f8cf5733b7a2c0d, scope.system=all) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:22.530854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:22.532501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0013fd9d-93d1-4adc-80a9-c72e75267823 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:22.553275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 166/334] 10.222.0.18 () {56 vars in 1308 bytes} [Sun Jul 24 00:47:22 2022] GET /identity/v3/role_assignments?user.id=435e89328cab4cfc9f8cf5733b7a2c0d&scope.system=all&role.id=881cc16b876541819935dfa513a16e5d => generated 510 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:24.013424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a8aaf-0d0c-49ff-8674-8e46e9730229 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.013941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a8aaf-0d0c-49ff-8674-8e46e9730229 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.014017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a5a8aaf-0d0c-49ff-8674-8e46e9730229 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.015643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 169/335] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:24.022132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c122b0-fc6d-43b2-bcc1-1512ac4cfa73 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.022595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c122b0-fc6d-43b2-bcc1-1512ac4cfa73 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.022805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c122b0-fc6d-43b2-bcc1-1512ac4cfa73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.041155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-81c122b0-fc6d-43b2-bcc1-1512ac4cfa73 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:24.042473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-81c122b0-fc6d-43b2-bcc1-1512ac4cfa73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:24.130437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 167/336] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:24.154824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe4020-ff26-4f1a-bda6-558d83bace6d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.155127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe4020-ff26-4f1a-bda6-558d83bace6d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.155397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ebe4020-ff26-4f1a-bda6-558d83bace6d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.156916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 170/337] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:24 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:24.162348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483ec2f-6aa1-43ac-bdc0-454420c420f3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.162638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483ec2f-6aa1-43ac-bdc0-454420c420f3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.162975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c483ec2f-6aa1-43ac-bdc0-454420c420f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.175836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebe4020-ff26-4f1a-bda6-558d83bace6d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:24.193562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c483ec2f-6aa1-43ac-bdc0-454420c420f3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:24.194752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c483ec2f-6aa1-43ac-bdc0-454420c420f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:24.348792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 168/338] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 187 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:24.366866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c98db-f6ea-43de-be31-ab22a542796b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.367176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c98db-f6ea-43de-be31-ab22a542796b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.367432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-656c98db-f6ea-43de-be31-ab22a542796b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.369629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 169/339] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:24 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:24.375003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.375287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.375561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.392388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ebe4020-ff26-4f1a-bda6-558d83bace6d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:24.396404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:24.406508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:24.407741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:24.462057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:24.463433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.463694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:24.463830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.463964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] PATH_INFO: `/v3/roles/reader` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.490096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] Could not find role: reader.: keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:24.504634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:24.505541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: reader. Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:24.506307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 171/340] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:24 2022] GET /identity/v3/roles/reader => generated 84 bytes in 330 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:24.517605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:24.547332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 170/341] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:24 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:24.551744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:24.698584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fefdd6de-ccfc-4298-b0bc-326e708653bd None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:24.720188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9dc56a1-37ed-47f5-8cab-7c1f3622e0dc None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:24.765807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:24.766484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.766784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.767070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.767824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] RBAC: Adding query filter params (name=reader) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:24.768135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:24.769519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:24.794414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:24.795285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:24.795829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:24.796113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:24.796654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 171/342] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:24 2022] GET /identity/v3/roles?name=reader => generated 341 bytes in 240 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:24.799012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:24.822995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:24.831438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:24.832310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:24.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:24.840879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 172/343] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:24 2022] GET /identity/v3/roles/member => generated 84 bytes in 334 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:24.842230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:24.863848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:24.879425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:24.992677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-228195a9-5849-49f5-ada4-e17e64332a85 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.015868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb7a0b6c-a442-4fc3-ba11-bcb45e58c86c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.061801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.062691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.062998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.063270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] PATH_INFO: `/v3/users/c5c0af3abdd245cdae0e726530bd1937` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.065139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] RBAC: Authorizing `identity:get_user(user_id=c5c0af3abdd245cdae0e726530bd1937)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:25.066603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:25.068259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 173/344] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:47:24 2022] GET /identity/v3/users/c5c0af3abdd245cdae0e726530bd1937 => generated 293 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:25.069007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.085670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.086488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.086786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.087065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.119239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:25.127686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:25.128600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 172/345] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:24 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:25.130655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.210066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.226413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88fa5d4d-a735-4731-987b-4cb6bd8a1803 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.244779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.258335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2bb5eda-ca4e-4193-8068-122ef482add8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.273447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.274460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.275502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.275980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.276935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:25.277464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:25.279083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1861c28e-acf0-4936-83f0-43109f769089 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:25.288637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 174/346] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 220 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:25.290200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50c7a6-ab84-49ac-aa36-9bb940a73bb4 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.290904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50c7a6-ab84-49ac-aa36-9bb940a73bb4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.291409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd50c7a6-ab84-49ac-aa36-9bb940a73bb4 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.292938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 175/347] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:25.293828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50c7a6-ab84-49ac-aa36-9bb940a73bb4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.303364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.304093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.304388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.304670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.305560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] RBAC: Adding query filter params (role.id=b65cce2fe3734269906540559fc82939, user.id=c5c0af3abdd245cdae0e726530bd1937, scope.system=all) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:25.305881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:25.308164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:25.326931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 173/348] 10.222.0.18 () {56 vars in 1308 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/role_assignments?user.id=c5c0af3abdd245cdae0e726530bd1937&scope.system=all&role.id=b65cce2fe3734269906540559fc82939 => generated 510 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:25.327997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.489329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd50c7a6-ab84-49ac-aa36-9bb940a73bb4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.493086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29d90c29-ce54-4421-b858-ed16d2e62401 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.562329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.562942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.563230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.563511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.565506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.566176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.566456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.566733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.567508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:25.567811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:25.569204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93a65db1-8920-4493-8b3b-f9ba7263bdbe None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:25.585116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 176/349] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 289 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:25.586753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.588497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.589661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.594330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.620780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:25.623393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:25.643446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.680519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:25.682602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb9ae7b9-ded3-49ac-ac64-e36426ce1a00 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:25.739374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 177/350] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:25 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 153 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:25.740446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.805760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 174/351] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 478 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:25.808048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2618688-6a3f-4c76-8358-1a07cd2950b5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.808513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2618688-6a3f-4c76-8358-1a07cd2950b5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.808790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2618688-6a3f-4c76-8358-1a07cd2950b5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.810164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 175/352] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:25.811291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2618688-6a3f-4c76-8358-1a07cd2950b5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:25.857247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3431ae5b-3dc1-4c63-ba87-4ac50d2f2ed6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.869225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.910303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.912764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.912764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.912764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:25.933173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:25.947358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2618688-6a3f-4c76-8358-1a07cd2950b5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:25.990550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:25.991432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:25.991721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:25.992005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.005260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:26.005459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.006866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53a2b359-e910-4bad-894f-f968a4ba3886 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:26.014985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:26.015999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:26.016784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:26.018815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 176/353] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/roles/member => generated 84 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:26.019777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.143125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 178/354] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:25 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 403 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:26.145020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.145309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.145585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.156420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6887564-abb5-49f1-bdd7-715078c46d4e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.163651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.163804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:26.165271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:26.208554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.209428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.209711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.209987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.211355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.213593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.215920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 177/355] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:26.216811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.229925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.263995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 179/356] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:26.265172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.333747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3df5d1f-63a8-41ec-a32e-9d9d38d1b335 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.378194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8657713e-0006-4413-9918-45d709a57194 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.378194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657713e-0006-4413-9918-45d709a57194 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.378454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657713e-0006-4413-9918-45d709a57194 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.378706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8657713e-0006-4413-9918-45d709a57194 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.381857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8657713e-0006-4413-9918-45d709a57194 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:26.381857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8657713e-0006-4413-9918-45d709a57194 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.381857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8657713e-0006-4413-9918-45d709a57194 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.382102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-763de1ad-6af7-463b-adde-fb494b12ffc5 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.390521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 178/357] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:26.391392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8657713e-0006-4413-9918-45d709a57194 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.431630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.432778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.433058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.433327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.434457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.434578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.436296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d239265-0de9-4191-91b5-88bd66ad2e3a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.438170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 180/358] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:26.440138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e96317-2da7-4b3f-800b-c5b99067202a None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.440425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e96317-2da7-4b3f-800b-c5b99067202a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.440702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89e96317-2da7-4b3f-800b-c5b99067202a None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.441949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 181/359] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:26.442819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e96317-2da7-4b3f-800b-c5b99067202a None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.500157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.520947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8657713e-0006-4413-9918-45d709a57194 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.565707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.566571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.566883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.566983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89e96317-2da7-4b3f-800b-c5b99067202a None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.569899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.573992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.575708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.577415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 179/360] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:26.582953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.610673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.611526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.611807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.612075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.625136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:26.633571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:26.634638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:26.635614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 182/361] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:26.636271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.703566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.707072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954e8309-042f-441f-8bdf-a6a9d2ef1721 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.755882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.756548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.756835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.757738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.762954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-daca67d7-ea09-4ab3-a789-8a9114df86ea None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:26.765319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.801846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:26.802723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.803045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.803752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.804614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.806470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd34fd79-bd5f-4107-acb6-654988a8ad38 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.808504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 183/362] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:26.809905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:26.810194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:26.810470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:26.829900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:26.831166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:26.841662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:26.843880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:26.914096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 184/363] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:26 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:26.915140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:26.952581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:26.956574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 180/364] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 379 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:26.957489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.043641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deed431b-2065-4142-9814-72f09bbbd878 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.045953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.075759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9a3f10-835a-4594-814d-ff089fe15596 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.082109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.082864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.083079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.083352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.084176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:27.084488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.085986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.098888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 185/365] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=881cc16b876541819935dfa513a16e5d => generated 260 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.100254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.116068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.116760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.117042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.117315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.118309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:27.118465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.119969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.138970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 181/366] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:26 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.138970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.224597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.236182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dab4a38-c647-4448-8728-0caa467f81a8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.278632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-805155e1-9075-43e2-b6ea-db77ff4b3250 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.296379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.297151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.297151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.297398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.298489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:27.299272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.300742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.307979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 186/367] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=496aed2e00104d4480f0efe375a79409 => generated 260 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.308934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.328952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.330900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.331446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.332203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.348869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:27.351499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:27.363338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:27.364303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:27.365091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:27.368862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 182/368] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/roles/admin => generated 83 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:27.375559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.459975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1cbd15-30e7-451c-a7f2-ed5cf2217d15 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.509908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.510715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.511027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.511305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.512663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.514127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.516303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 187/369] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.517050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.523939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24081eaa-eda3-429a-ae6d-6902f07752e7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.549233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.563956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.564722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.565077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.565430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.645107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.649527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.651412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.651753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5865ef2e-5973-41a2-bb25-ddc9307c68fd None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.701347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.701820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.702201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.702576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.703470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:27.703872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.706278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.717928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 188/370] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.719191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.777168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 183/371] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 406 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:27.778365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.807380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:27.866432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc3efb1-d91e-407e-b066-d2828179d3c2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.906812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763c912-204f-498b-8b72-86531116f407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.907043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-745ed9bc-3eac-4868-aeda-066b529c9810 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:27.908169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763c912-204f-498b-8b72-86531116f407 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.908656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763c912-204f-498b-8b72-86531116f407 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.909131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8763c912-204f-498b-8b72-86531116f407 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.910582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763c912-204f-498b-8b72-86531116f407 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:27.912282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8763c912-204f-498b-8b72-86531116f407 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:27.919277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 189/372] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 201 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:27.920432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763c912-204f-498b-8b72-86531116f407 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:27.948112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:27.948768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:27.949041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:27.949317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:27.964240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:28.028534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.030310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.058596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:28.068215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8763c912-204f-498b-8b72-86531116f407 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.113040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.114165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.114459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.115630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.116851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.118357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.120464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 190/373] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.121068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.158997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 184/374] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:27 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 381 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:28.160203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.247661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:28.256002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfbb0935-dc48-4248-968e-af8e367ea93d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.286189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a208da-720c-4aef-b6e0-915a7e21391c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.297565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.298633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.299156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.299635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.300736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=7d8f3d187861468eb86fbc334ae08015) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:28.301280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.303020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.310367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 191/375] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=881cc16b876541819935dfa513a16e5d => generated 260 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.314923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:28.316499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.330002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.330928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.331201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.331473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.332697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.334802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.337967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 185/376] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.344271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.502131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fcd3afa-634e-4dff-92d3-cbe27a2e537d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.557056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.557531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.557819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.558209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.559075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=7d8f3d187861468eb86fbc334ae08015) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:28.559389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.566796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.570447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 186/377] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=496aed2e00104d4480f0efe375a79409 => generated 260 bytes in 227 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.598376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6239e7e-795d-4371-b554-c855bf2f3cb8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.657502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.658854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.659394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.659884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] PATH_INFO: `/v3/projects/17ae0b4181524f419070ad7610eaf309` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.661339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] RBAC: Authorizing `identity:get_project(project_id=17ae0b4181524f419070ad7610eaf309)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.663104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.667628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 192/378] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/projects/17ae0b4181524f419070ad7610eaf309 => generated 344 bytes in 352 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.675929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.763657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:28.813786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c81b09-e602-457b-92d5-2a18524ccf63 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:28.850535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:28.851432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:28.851711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:28.851981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] PATH_INFO: `/v3/groups/f085871e549541858df5dddc616e15dd` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:28.858431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] RBAC: Authorizing `identity:get_group(group_id=f085871e549541858df5dddc616e15dd)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:28.859945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd84adae-2c11-440c-b320-3d54a5e9df85 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:28.862095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 187/379] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/groups/f085871e549541858df5dddc616e15dd => generated 229 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:28.871716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:28.917284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:29.058416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb28a977-af8e-4c0f-b46a-83f555682a88 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:29.124955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:29.125318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.125575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.125849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.126681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] RBAC: Adding query filter params (group.id=f085871e549541858df5dddc616e15dd, role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=17ae0b4181524f419070ad7610eaf309) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:29.127462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:29.129278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebbb6eff-c20c-4258-a2f2-e9a029598e72 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:29.139144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 193/380] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:28 2022] GET /identity/v3/role_assignments?group.id=f085871e549541858df5dddc616e15dd&scope.project.id=17ae0b4181524f419070ad7610eaf309&role.id=75e91b1e89304179904cf7644ef0a984 => generated 260 bytes in 269 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:29.535580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b634dfb-cd4b-4aca-a9b7-18d304a46add None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.536558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b634dfb-cd4b-4aca-a9b7-18d304a46add None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.537041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b634dfb-cd4b-4aca-a9b7-18d304a46add None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.538884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 188/381] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:29 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:29.544171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d7a4e0-6946-430e-b11e-58a66c15fb1e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.544570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d7a4e0-6946-430e-b11e-58a66c15fb1e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.544732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23d7a4e0-6946-430e-b11e-58a66c15fb1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.568035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-23d7a4e0-6946-430e-b11e-58a66c15fb1e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:29.569210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-23d7a4e0-6946-430e-b11e-58a66c15fb1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:29.668541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 194/382] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:29.716657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b634dfb-cd4b-4aca-a9b7-18d304a46add None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:29.845807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b634dfb-cd4b-4aca-a9b7-18d304a46add None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:29.865466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:29.893423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:29.894598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.895108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.895577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.901505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb000f1-a781-4fea-ab27-66309a2e81c0 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.901796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb000f1-a781-4fea-ab27-66309a2e81c0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.902077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb000f1-a781-4fea-ab27-66309a2e81c0 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.906791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 195/383] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:29.909079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:29.909189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:29.909453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:29.912510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:29.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:29.921673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:29.922459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:29.925254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 189/384] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:29 2022] GET /identity/v3/roles/member => generated 84 bytes in 209 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:29.932334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:29.955765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:29.956982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:30.004286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:30.077842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62e672e0-8c81-4906-92cc-609bb4f7a0f5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.103853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 196/385] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:29 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:30.121794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.122824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.123308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.123777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.124719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:30.125218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.126093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.127073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.143089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 190/386] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:29 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:30.145015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.194262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:30.278434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7409f340-0681-4540-84d3-89ecbd937527 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.301139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c948852-299d-4d00-bbb4-82d4f46e4cb5 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.331924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.332783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.333064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.333339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.336724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.339045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.340880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 191/387] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:30.346719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.381045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.381862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.382141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.382414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.406644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:30.423729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:30.424756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:30.425609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:30.426521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 197/388] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 301 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:30.435993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.479285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:30.502589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288e085e-371a-4de1-bfc4-2b90a0ad9ed7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.512918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:30.542371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.543988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.544852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.545299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.551676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.553332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.557373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 192/389] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 211 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:30.572622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.655622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd03c85f-71d7-471b-be46-5c5c2503b841 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.729946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.730552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.730862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.731143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.731899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:30.732207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.733599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.753103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 198/390] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 318 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:30.759389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.782647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53c2ba77-99bb-4d7f-a94a-f4f9ce239fe1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.850410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:30.871519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.872427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.872716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.873000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/groups/d630b26e769545e682d32d9a0035e41d/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.875063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, group_id=d630b26e769545e682d32d9a0035e41d, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.876514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.880634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4045ed9-ed58-4752-b1c4-9f7424eb402a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:30.906090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 193/391] 10.222.0.18 () {58 vars in 1442 bytes} [Sun Jul 24 00:47:30 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/groups/d630b26e769545e682d32d9a0035e41d/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 334 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:30.920799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:30.921781 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.922136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.922519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.923947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:30.925551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5bbac21-ff9b-48bd-952c-f5827ed2105f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:30.927721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 199/392] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:30.933644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:30.964640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7e9a-6ca8-4a09-af06-575ce8c29b2a None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.964979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7e9a-6ca8-4a09-af06-575ce8c29b2a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.965223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f47e7e9a-6ca8-4a09-af06-575ce8c29b2a None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.967253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 200/393] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:30.972611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:30.973015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:30.973765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:30.995808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:30.997132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:31.016867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.117184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 201/394] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:30 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:31.128203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84d30b40-0674-4f68-baaa-cb53d43d6487 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.141404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.174976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.174976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.174976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.174976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.175395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:31.175395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:31.186042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 194/395] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:30 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:31.189598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.263950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.298865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e97319ae-47c3-473a-9618-b2009178aea3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.336924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.338624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.338953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90bf076d-3015-4b94-9ff3-366d8e3859a7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.339308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.339758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.341526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.351882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-354fd90b-0962-4e7c-89d2-9bb482c1df28 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:31.360787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:31.361858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:31.362650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:31.365333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 202/396] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/roles/member => generated 84 bytes in 224 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:31.366743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b325df-6a2d-4358-8d81-79345e7d108c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.367240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b325df-6a2d-4358-8d81-79345e7d108c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.367745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04b325df-6a2d-4358-8d81-79345e7d108c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.369508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 203/397] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:31.371915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b325df-6a2d-4358-8d81-79345e7d108c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.386093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.387425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.387711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.387990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/groups/d630b26e769545e682d32d9a0035e41d/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.390735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] RBAC: Authorizing `identity:create_grant(project_id=6f3cc53c4c724b8fbbf2a5007a00648c, group_id=d630b26e769545e682d32d9a0035e41d, role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:31.393435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d812324f-565b-43d1-88c9-e715a10faba8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:31.417073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 195/398] 10.222.0.18 () {58 vars in 1442 bytes} [Sun Jul 24 00:47:31 2022] PUT /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c/groups/d630b26e769545e682d32d9a0035e41d/roles/496aed2e00104d4480f0efe375a79409 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:31.430379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.430379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.430379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.446680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:31.450720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:31.517567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04b325df-6a2d-4358-8d81-79345e7d108c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.550992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 196/399] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:31.563179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.570501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.571738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.572127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.572464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.573296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] RBAC: Adding query filter params (name=member) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:31.573658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:31.575293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:31.579952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.590514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 204/400] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:31.596858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.732597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98155d06-08f0-4b04-960f-d417dee5a22b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.740970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.758114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e566f755-a3e7-4c29-a2ef-699287c6ec64 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:31.784841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.785622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.785900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.786172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.787445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:31.788881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a733866d-a1c6-4711-9293-76501bfffb59 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:31.791318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 205/401] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:31.792491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec453ca1-8439-4d05-9636-e367ba1492b0 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.792771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec453ca1-8439-4d05-9636-e367ba1492b0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.793041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec453ca1-8439-4d05-9636-e367ba1492b0 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.798997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 206/402] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:31 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:31.799739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec453ca1-8439-4d05-9636-e367ba1492b0 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.802579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:31.803636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.804116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.804587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.819286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-863e6419-67a0-4156-aeaa-cbd7a71c1d08 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:31.827866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:31.828800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:31.829575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 197/403] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:31.830713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:31.830984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:31.831270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:31.849226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:31.850399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:31.892010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:31.923277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 198/404] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:31 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:31.924123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:31.943219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec453ca1-8439-4d05-9636-e367ba1492b0 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.015865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.017336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.017819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.018267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.021394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.026459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.029581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 207/405] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:32.037949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.042092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:32.061350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b79e9181-7521-4a2f-8773-13b9d825adeb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.098311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.099422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.099913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.100458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.101508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:32.102029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.104449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.113617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 199/406] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:31 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:32.114670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.213869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec4f48fb-166c-4721-a532-b1178357950d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.226108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3f1186c-fa0c-4715-83e5-9a72bae41ceb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.263721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:32.264686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.266610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.267359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.267893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/groups/d630b26e769545e682d32d9a0035e41d/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.271610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, group_id=d630b26e769545e682d32d9a0035e41d, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.273234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.273613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.275152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.275642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.276135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.290963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 200/407] 10.222.0.18 () {58 vars in 1442 bytes} [Sun Jul 24 00:47:32 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/groups/d630b26e769545e682d32d9a0035e41d/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:32.292436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.303525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:32.307745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:32.332943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:32.333985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:32.334808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:32.343430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 208/408] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:32 2022] GET /identity/v3/roles/admin => generated 83 bytes in 313 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:32.352255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.422963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76400250-f08c-4ba0-ad53-10a8fe4f45b6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.462303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.463813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.464387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.464884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.466298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.468002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.469937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 201/409] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:32 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:32.479563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.587477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9659d4bd-0c18-47aa-8685-adda9088c46f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.639102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:32.654584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.655723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.656205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.656689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.657998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:32.658516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.662181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.672225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fd4ddbe-b1b7-4af6-9e2f-cf8deb383b44 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.678861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 209/410] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:32 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 327 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:32.687576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.713189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.714127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.714420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.714700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.716404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.718195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.720544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 202/411] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:32 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 242 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:32.726184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:32.834289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff52719a-0beb-455d-8b84-de93dd7c4bce None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:32.864430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:32.872493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:32.873389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:32.873684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:32.873966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015/groups/d630b26e769545e682d32d9a0035e41d/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:32.876533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] RBAC: Authorizing `identity:create_grant(project_id=7d8f3d187861468eb86fbc334ae08015, group_id=d630b26e769545e682d32d9a0035e41d, role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:32.877993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:32.909239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 203/412] 10.222.0.18 () {58 vars in 1442 bytes} [Sun Jul 24 00:47:32 2022] PUT /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015/groups/d630b26e769545e682d32d9a0035e41d/roles/496aed2e00104d4480f0efe375a79409 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:32.967761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0a2d28-957d-4062-a825-06b634952ac4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:33.031417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:33.032898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:33.033390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:33.033884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] PATH_INFO: `/v3/groups/f085871e549541858df5dddc616e15dd` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:33.036082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] RBAC: Authorizing `identity:get_group(group_id=f085871e549541858df5dddc616e15dd)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:33.038727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:33.042335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 210/413] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:32 2022] GET /identity/v3/groups/f085871e549541858df5dddc616e15dd => generated 229 bytes in 356 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:33.050137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:33.057507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:33.179287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d116bbd-c4a6-4942-aacd-c5a3dfa95642 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:33.215245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:33.216121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:33.216407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:33.216687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] PATH_INFO: `/v3/projects/17ae0b4181524f419070ad7610eaf309` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:33.217918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] RBAC: Authorizing `identity:get_project(project_id=17ae0b4181524f419070ad7610eaf309)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:33.219403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8b3d1ca-a9a0-4c3c-8753-74db6f5e4448 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:33.221433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 204/414] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:33 2022] GET /identity/v3/projects/17ae0b4181524f419070ad7610eaf309 => generated 344 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:33.226872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:33.370050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9857ba7a-8db4-4be3-b3b6-26d885e8947c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:33.407146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:33.410601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:33.411545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:33.411932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:33.412193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] PATH_INFO: `/v3/projects/17ae0b4181524f419070ad7610eaf309/groups/f085871e549541858df5dddc616e15dd/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:33.414435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] RBAC: Authorizing `identity:create_grant(project_id=17ae0b4181524f419070ad7610eaf309, group_id=f085871e549541858df5dddc616e15dd, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:33.415990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f732668-3130-48bf-985a-5d1669940d08 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:33.438554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 211/415] 10.222.0.18 () {58 vars in 1442 bytes} [Sun Jul 24 00:47:33 2022] PUT /identity/v3/projects/17ae0b4181524f419070ad7610eaf309/groups/f085871e549541858df5dddc616e15dd/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 212 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:47:33.771710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab9abe0-d656-44c2-9e64-ebbb0dfb3d1b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:33.772212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab9abe0-d656-44c2-9e64-ebbb0dfb3d1b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:33.772212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ab9abe0-d656-44c2-9e64-ebbb0dfb3d1b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:33.773769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 205/416] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:33.779075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea67822-7256-4ac6-a89f-1dbc64d33d60 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:33.779370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea67822-7256-4ac6-a89f-1dbc64d33d60 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:33.779649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ea67822-7256-4ac6-a89f-1dbc64d33d60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:33.797679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3ea67822-7256-4ac6-a89f-1dbc64d33d60 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:33.798907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3ea67822-7256-4ac6-a89f-1dbc64d33d60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:33.891677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 212/417] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:33 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:33.927571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab9abe0-d656-44c2-9e64-ebbb0dfb3d1b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.026492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:34.076685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ab9abe0-d656-44c2-9e64-ebbb0dfb3d1b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:34.080553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52c51d-1fdd-43b7-8736-7b6888795332 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.081070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52c51d-1fdd-43b7-8736-7b6888795332 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.081540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c52c51d-1fdd-43b7-8736-7b6888795332 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.083121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 213/418] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:34.088780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.089304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.089805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.115290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:34.115290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.115290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.115290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.122273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:34.130815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:34.132324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:34.135518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:34.136386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:34.137194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:34.139598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 206/419] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:33 2022] GET /identity/v3/roles/member => generated 84 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:34.147130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.264570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 214/420] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:34 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:34.286469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d66db12-ab85-4100-ab6c-668e43d1a631 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:34.300946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.338664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:34.348577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:34.349271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.349556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.349837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.350597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:34.350935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:34.352326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:34.377102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 207/421] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 230 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:34.382211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.386105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:34.512891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad6e0559-9513-49bf-9715-d501701d9ec9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:34.586587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:34.587899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.588203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.588570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.606970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:34.615259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:34.616225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:34.617034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 215/422] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 316 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:34.623398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0970eec3-3a4a-47fb-bc17-a000fabbc159 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:34.634838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.684574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:34.685236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.685522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.686054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.769354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:34.779207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:34.780790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:34.828959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0a2b8fd-fee0-463f-9bee-8c2fd571d5ed None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:34.831312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:34.880593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:34.881190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:34.881472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:34.881758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:34.882513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:34.882843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:34.884352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:34.907525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 216/423] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 273 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:34.909064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:34.946648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 208/424] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 565 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:47:34.952242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.076145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.094074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-273a64e6-7015-4823-8eeb-74b136aa3149 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.118003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df11ab6-ef5f-4ef7-8cc5-4ee26e1d762d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.162607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.163736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.164227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.164646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.166745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.168445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.175433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 209/425] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 223 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:35.186820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.186982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.186982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.186982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.187225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.222329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.277162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.279379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68568f0f-aad5-48cb-b467-cd21bfd24e9f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.320292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60fa708c-9e8a-43fd-a704-a59333d4d2a6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.359759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.361617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.362111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.362584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.363980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.366144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87b98ce2-696e-40a5-ab5c-e85355237137 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.369617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 210/426] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:35.371407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff150df-0cd6-42c3-8b5e-a9219b135c66 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.372029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff150df-0cd6-42c3-8b5e-a9219b135c66 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.372829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff150df-0cd6-42c3-8b5e-a9219b135c66 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.374570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 211/427] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:35.376347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae55d7ef-8bcb-44b7-8011-ff587aab97e0 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.376817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae55d7ef-8bcb-44b7-8011-ff587aab97e0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.377288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae55d7ef-8bcb-44b7-8011-ff587aab97e0 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.379245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 212/428] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:35 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:35.380149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae55d7ef-8bcb-44b7-8011-ff587aab97e0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.396010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 217/429] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:34 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 487 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:47:35.397402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6403b244-8055-4c92-ade8-856e85273b88 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.397691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6403b244-8055-4c92-ade8-856e85273b88 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.397972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6403b244-8055-4c92-ade8-856e85273b88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.414105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6403b244-8055-4c92-ade8-856e85273b88 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:35.415852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6403b244-8055-4c92-ade8-856e85273b88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:35.422962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.478527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.491250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 218/430] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:35.492675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.493160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.493939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.518694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae55d7ef-8bcb-44b7-8011-ff587aab97e0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.519482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:35.520665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:35.555123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.556542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.556840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.557123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.557949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:35.558261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.559718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.570809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 213/431] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=881cc16b876541819935dfa513a16e5d => generated 611 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:35.570809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.607253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 219/432] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:35 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:35.608065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.714691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b552b521-fb4e-466f-8925-6cb921e045dc None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.719738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.723479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96b6bc61-9d2f-4bf1-96cc-689c1d4a83e4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.754168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:35.759650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.760686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.760807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.761081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] PATH_INFO: `/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.763366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] RBAC: Authorizing `identity:get_project(project_id=6f3cc53c4c724b8fbbf2a5007a00648c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.764833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.770324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 214/433] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/projects/6f3cc53c4c724b8fbbf2a5007a00648c => generated 298 bytes in 200 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:35.770324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.777305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.778364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.778648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.779451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] PATH_INFO: `/v3/roles/member` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.794103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] Could not find role: member.: keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:35.803190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:35.804143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: member. Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:35.805087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 220/434] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/roles/member => generated 84 bytes in 197 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:35.812614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.889308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08ff9928-1caf-4181-9c37-ed333b815b62 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.933915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.934928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.935218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.935498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.939767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-571de0e6-7799-4d82-96d2-951250023f50 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:35.953154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:35.965162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:35.966243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:35.967448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:35.968662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 215/435] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:35.971518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.985450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:35.986495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:35.986924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:35.987300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:35.989671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:35.991398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:35.994158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 221/436] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 189 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:35.995081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:35.999951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.054615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.083627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849ea837-45d6-4515-a119-96a326220dbf None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.105467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba942a82-7b91-4a0b-90de-e2518b122c64 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.116447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.117112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.117400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.117676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.118444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] RBAC: Adding query filter params (name=member) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:36.118975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.120357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.128151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 216/437] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/roles?name=member => generated 341 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.128931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.139753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.140425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.140711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.140990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.141742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:36.142051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.143442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ecfb694-0014-4264-98e0-edb324402302 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.152093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 222/438] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:35 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.153292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a0411-8269-4860-8faa-4626d106c76e None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.153582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a0411-8269-4860-8faa-4626d106c76e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.153865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-246a0411-8269-4860-8faa-4626d106c76e None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.155286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 223/439] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:36 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:36.155810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246a0411-8269-4860-8faa-4626d106c76e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.213237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.232098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92b559c8-88d1-4d21-bd6c-38f129351a1b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.256347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-246a0411-8269-4860-8faa-4626d106c76e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.265532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.266189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.266474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.266992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.267683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:36.267914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.269339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.269441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.275766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 217/440] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=6f3cc53c4c724b8fbbf2a5007a00648c&role.id=496aed2e00104d4480f0efe375a79409 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.276563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.300179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.301200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.301634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.302061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.385447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.387794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d3777c8-1789-42ad-bdc3-2c31b72e526a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.395105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce2a9e3-c1c0-4a65-a279-17a349c60769 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.440333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.441018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.441286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.441565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.459430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.505051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 224/441] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 349 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:36.506650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.507140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.507639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.520741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.522218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.528871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:36.530068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:36.537801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.617258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 225/442] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:36 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:36.618586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.646915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 218/443] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:36.648139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.699920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.726148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ef2cfe1-8f3c-48b4-afd5-5c55d464944f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.753192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df0e77ea-7a5b-4d33-9987-02b6d037f610 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.760054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.760848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.761129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.761405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.762585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.764061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.765741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 226/444] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.767878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.781349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.789078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.789894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.790258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.790611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.800465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:36.808251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:36.809130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:36.809941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 219/445] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 162 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:36.810907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.873033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1ffa352-b3e7-4c87-b6fb-0968d46174ce None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.907959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.908614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78b1f59-af6c-4757-b59d-e9e5bd503c3b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:36.909156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.909156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.909401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] PATH_INFO: `/v3/projects/7d8f3d187861468eb86fbc334ae08015` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.910503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] RBAC: Authorizing `identity:get_project(project_id=7d8f3d187861468eb86fbc334ae08015)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.911983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.913868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 227/446] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/projects/7d8f3d187861468eb86fbc334ae08015 => generated 302 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.914512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.915316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:36.944690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:36.945534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:36.945819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:36.946082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:36.947847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:36.949293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:36.950914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 220/447] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:36.951665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:36.993254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.015657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8d1fefd-70a8-4edb-8004-09cd94f06a4c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.048467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21d611d3-9f6b-4de1-90d6-7513f811e5fd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.048870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.049534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.049819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.050093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.050863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:37.051172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.052543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.065149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 228/448] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.065149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.081815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.082668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.082983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.083259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] PATH_INFO: `/v3/groups/d630b26e769545e682d32d9a0035e41d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.084421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] RBAC: Authorizing `identity:get_group(group_id=d630b26e769545e682d32d9a0035e41d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.085853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.088217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 221/449] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:36 2022] GET /identity/v3/groups/d630b26e769545e682d32d9a0035e41d => generated 226 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.088886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.132197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.165644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa765be1-c3f1-4e02-bbc1-85538a28528f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.188086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-078d5bf9-464e-4dee-9e6b-bdc86d727c0e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.199857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.200513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.200795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.201072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.201898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=881cc16b876541819935dfa513a16e5d, scope.project.id=7d8f3d187861468eb86fbc334ae08015) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:37.202210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.203703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.209522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 229/450] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=881cc16b876541819935dfa513a16e5d => generated 611 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.210294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.213730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.233716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.234668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.235002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.235290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.313987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.315428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.329269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02f79ea7-ed26-4f06-beea-24cee18295f9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.360009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.368644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.369176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.369359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.369637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.370761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] RBAC: Adding query filter params (group.id=d630b26e769545e682d32d9a0035e41d, role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=7d8f3d187861468eb86fbc334ae08015) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:37.370761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.372279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.379618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 230/451] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/role_assignments?group.id=d630b26e769545e682d32d9a0035e41d&scope.project.id=7d8f3d187861468eb86fbc334ae08015&role.id=496aed2e00104d4480f0efe375a79409 => generated 611 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.438693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 222/452] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:37.444400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.523596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.560514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f34390c-0416-4d84-b0f3-9f955b54c3ec None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.596608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.597393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.597669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.597938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] PATH_INFO: `/v3/projects/17ae0b4181524f419070ad7610eaf309` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.599236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] RBAC: Authorizing `identity:get_project(project_id=17ae0b4181524f419070ad7610eaf309)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.600686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.603065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 231/453] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/projects/17ae0b4181524f419070ad7610eaf309 => generated 344 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.608654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.709462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c8e3d4-f0b0-4dd9-8775-cfc2d07a6c9f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.744061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.744925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.745214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.745499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] PATH_INFO: `/v3/groups/f085871e549541858df5dddc616e15dd` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.746680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] RBAC: Authorizing `identity:get_group(group_id=f085871e549541858df5dddc616e15dd)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.748163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f20c4b69-6558-4ebc-8c75-fa492c0f659b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.749791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 223/454] 10.222.0.18 () {56 vars in 1180 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/groups/f085871e549541858df5dddc616e15dd => generated 229 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:37.755706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:37.855968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee0b7c5c-21eb-4319-826b-57a295406ef7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:37.890544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:37.891208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:37.891494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:37.891773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:37.892601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] RBAC: Adding query filter params (group.id=f085871e549541858df5dddc616e15dd, role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=17ae0b4181524f419070ad7610eaf309) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:37.892916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:37.894347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf9be0db-f8b6-433a-b24b-8a6cb63f65b3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:37.897144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:37.902196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 232/455] 10.222.0.18 () {56 vars in 1376 bytes} [Sun Jul 24 00:47:37 2022] GET /identity/v3/role_assignments?group.id=f085871e549541858df5dddc616e15dd&scope.project.id=17ae0b4181524f419070ad7610eaf309&role.id=75e91b1e89304179904cf7644ef0a984 => generated 611 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:40.504224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244404b3-0f2f-4533-ab8c-6d380d608fcb None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.505159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244404b3-0f2f-4533-ab8c-6d380d608fcb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.505624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-244404b3-0f2f-4533-ab8c-6d380d608fcb None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.507493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 224/456] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:40.513080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7076b959-745b-40d6-9365-a124b0ae0e38 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.513532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7076b959-745b-40d6-9365-a124b0ae0e38 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.513634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7076b959-745b-40d6-9365-a124b0ae0e38 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.535770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7076b959-745b-40d6-9365-a124b0ae0e38 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:40.537223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7076b959-745b-40d6-9365-a124b0ae0e38 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:40.627487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 233/457] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:40.645650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b690f71-17d6-4eb4-852b-9881fb6b8f40 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.645650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b690f71-17d6-4eb4-852b-9881fb6b8f40 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.645650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b690f71-17d6-4eb4-852b-9881fb6b8f40 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.646210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 225/458] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:40.650747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1565860-755a-4c4c-a751-00cdc559577e None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.651037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1565860-755a-4c4c-a751-00cdc559577e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.651267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1565860-755a-4c4c-a751-00cdc559577e None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.652835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.652962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 234/459] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:40 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:40.653456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.653960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.657785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.658069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.658345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.675040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:40.676794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:40.676908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:40.677964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:40.681693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:40.761815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 235/460] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:40.762920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:40.780497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 226/461] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:40 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:40.793341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:40.831907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:40.876379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d23abcf-885d-40e4-98db-0cf4d74855e2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:40.920811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:40.921707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.921989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.922262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.924109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbbb457e-ee61-460e-94e6-197b2b4783c0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:40.933923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:40.935938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:40.957048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:40.958098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:40.959181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:40.959181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:40.959181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:40.959181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:40.959461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:40.959461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.959461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.959461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.966594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 236/462] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:40.966594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:40.967821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:40.970383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:40.972264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-a4b89dac-dac4-4f8a-9eea-42fc259d6c97 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:40.992208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:40.993066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:40.993894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:40.993894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:40.993894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:40.993894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:40.993894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 227/463] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:40.994764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8b6887-d0ed-43ec-9e3e-11db320a0592 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:40.995108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8b6887-d0ed-43ec-9e3e-11db320a0592 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:40.995391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d8b6887-d0ed-43ec-9e3e-11db320a0592 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:40.998927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 228/464] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:40.998927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8b6887-d0ed-43ec-9e3e-11db320a0592 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.078908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce656c7-9463-4c41-960b-e12a0073ed34 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.092633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.114641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d8b6887-d0ed-43ec-9e3e-11db320a0592 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.117762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.118853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.119496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.120013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.134169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.135992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.151064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.151803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.152091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.152370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.153096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:41.153405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.154787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5c49dec-3cac-494a-8e18-c47e9ef9206c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:41.158328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:41.159333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:41.160441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:41.160441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:41.160441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:41.160441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:41.160441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 237/465] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:41.160895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.176122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 229/466] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:41.176122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.176122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.176122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.191917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:41.193036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:41.241566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.264718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 230/467] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:41 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:41.265890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.288348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de07cacd-fe1e-4d3f-86bf-3f11fcfd59c5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.325682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.326337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.326619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.326945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.327659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:41.327963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.329334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.341406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 238/468] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:41.342165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.369383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.379869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b953434-853c-4da3-a935-af7b5cc62b41 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.422935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.423607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.423893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.424167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.424892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:41.425193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.426577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.439145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 231/469] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:41.444188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.452521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c29fcf49-65b7-40f3-a6c3-b7ec573684bd None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.487692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.489861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.490917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.491638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.491989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.492692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.494273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.526802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 239/470] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:41 2022] POST /identity/v3/users => generated 246 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:41.526802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.575384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cdede843-83e1-46c7-ba43-f80d58c740e7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.610285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.632110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.632929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.633210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.633506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.642469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a34e1cc-1c44-405c-80da-0f22c21211f6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.651684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.653392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:41.679179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:41.680120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:41.681191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:41.681191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:41.681191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:41.681191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:41.683381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 232/471] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:41 2022] GET /identity/v3/domains/Default => generated 87 bytes in 244 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:41.684404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.687575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1599ecec-8142-40af-9409-593d743897ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.688662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1599ecec-8142-40af-9409-593d743897ff None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.688951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1599ecec-8142-40af-9409-593d743897ff None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.689229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1599ecec-8142-40af-9409-593d743897ff None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.690996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1599ecec-8142-40af-9409-593d743897ff None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.692464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1599ecec-8142-40af-9409-593d743897ff None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.717124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 240/472] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:41 2022] POST /identity/v3/users => generated 248 bytes in 200 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:41.718231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1599ecec-8142-40af-9409-593d743897ff None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:41.823706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.833729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-051bebcd-1e5f-4720-ab5c-e5abd42dcb08 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.864168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1599ecec-8142-40af-9409-593d743897ff None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:41.877794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.878918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.879408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.879902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.880818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.882467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.908218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 233/473] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:41 2022] POST /identity/v3/users => generated 248 bytes in 224 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:41.910831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:41.911525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:41.911814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:41.912094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:41.912844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:41.913161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:41.914755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333ed9f0-c475-476d-bc9d-aa8476f1eb69 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:41.924491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:41.931098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 241/474] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:41.938876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:42.052520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb037a-25ee-4409-8b90-b9f864eddabb None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.053078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb037a-25ee-4409-8b90-b9f864eddabb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.053531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0bb037a-25ee-4409-8b90-b9f864eddabb None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.055164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 242/475] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:42 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:42.059631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24a26b3e-2bc7-4b5c-a26b-0f004ee7adee None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:42.062211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.062718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.063218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.088695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:42.090344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:42.117085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:42.117769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.118296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.118587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.119394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:42.120858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:42.153215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 234/476] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:41 2022] POST /identity/v3/users => generated 249 bytes in 215 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:47:42.176048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 243/477] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:42 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:42.199805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:42.210476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:42.317870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71014f2d-e90d-4cf1-b6a8-977d45c95049 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:42.357859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:42.358818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.359243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.359659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.360577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] RBAC: Authorizing `identity:create_role()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:42.362082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:42.369018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.sql.core [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] Conflict role: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 24 00:47:42.369018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 24 00:47:42.369018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [parameters: {'id': '63a71fb0603b4172b54dae134e32ac37', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 24 00:47:42.369018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80232) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] Conflict occurred attempting to store role - Duplicate entry found with name anotherrole.: keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:47:42.386551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 33, in create_role Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return ref.to_dict() Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:47:42.387634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application yield resource Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.rollback() Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.force_reraise() Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise self.value Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application session.commit() Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 24 00:47:42.388777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._prepare_impl() Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.flush() Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._flush(objects) Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application compat.raise_( Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application flush_context.execute() Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rec.execute(self) Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 24 00:47:42.389832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:47:42.390973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:47:42.391900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'anotherrole-<>' for key 'role.ixu_role_name_domain_id'") Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [SQL: INSERT INTO `role` (id, name, description, extra) VALUES (%(id)s, %(name)s, %(description)s, %(extra)s)] Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [parameters: {'id': '63a71fb0603b4172b54dae134e32ac37', 'name': 'anotherrole', 'description': None, 'extra': '{}'}] Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:42.392780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 115, in post Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.role_api.create_role( Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1298, in create_role Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ret = self.driver.create_role(role_id, role) Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store role - Duplicate entry found with name anotherrole. Jul 24 00:47:42.393656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:42.396426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 235/478] 10.222.0.18 () {60 vars in 1131 bytes} [Sun Jul 24 00:47:42 2022] POST /identity/v3/roles => generated 144 bytes in 197 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:42.403122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:42.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91ac915b-7fd1-4dfe-9daa-e6350af88fb9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:42.547535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:42.557180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:42.558000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.558274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.558540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] PATH_INFO: `/v3/roles/anotherrole` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.569986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-2a3d45fa-56d8-4a0d-a849-40b4bf51ada3 None admin] Could not find role: anotherrole.: keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:42.579128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:42.580033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: anotherrole. Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:42.581020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 244/479] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:47:42 2022] GET /identity/v3/roles/anotherrole => generated 89 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:42.586009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:42.658835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:42.691940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d88d191-5395-4f6c-bb74-c8f415cfc407 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:42.726841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:42.728053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:42.728336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:42.728608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:42.729412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] RBAC: Adding query filter params (name=anotherrole) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:42.729717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:42.731152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de6c6eb9-285d-40d7-825e-2337bc7381d5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:42.739749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 236/480] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:42 2022] GET /identity/v3/roles?name=anotherrole => generated 334 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:44.080923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715e187-53fe-4a6c-8e99-cd2fd4a6e4d3 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.081454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715e187-53fe-4a6c-8e99-cd2fd4a6e4d3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.081454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a715e187-53fe-4a6c-8e99-cd2fd4a6e4d3 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.083115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 245/481] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:44.088568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.089012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.089192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.106300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:44.111540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:44.192876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c62b2-7dd2-4d74-b37f-f2da5df5d47c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.193542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c62b2-7dd2-4d74-b37f-f2da5df5d47c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.194020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6c62b2-7dd2-4d74-b37f-f2da5df5d47c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.197475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 246/482] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:44 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:44.201827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.202353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.202889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.203111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 237/483] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:44.220692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:44.222206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:44.239631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.326453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:44.354868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 247/484] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 149 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:44.376959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7292162d-56fa-458a-a31e-9b51fb67b0f0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.432649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.440830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.442045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.446748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.447268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.465161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-c842c3f2-3922-4c94-865d-a1d5bb1182db None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:44.479449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:44.480461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:44.481793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 238/485] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 242 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:44.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301df203-5f90-4ded-b6cf-564f868100c0 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301df203-5f90-4ded-b6cf-564f868100c0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.482443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301df203-5f90-4ded-b6cf-564f868100c0 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.483965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 239/486] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:44.488543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301df203-5f90-4ded-b6cf-564f868100c0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.570642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2af8ded7-a0fe-417a-97a3-485df038fc4d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.589546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:44.600340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301df203-5f90-4ded-b6cf-564f868100c0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.611076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.612284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.613054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.613537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.626011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:44.628020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-8340f300-fcf1-4a4b-9e0a-f040a21bcc6a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:44.635417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:44.636299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:44.637119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:44.639860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 248/487] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:44.641334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.641816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.642307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.642475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.643214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.643500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.643778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.644536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:44.644865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:44.646530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:44.656421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 240/488] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:44.657273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.662502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:44.663957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:44.737163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 249/489] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:44 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:44.738505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.783429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7bfd2f6-a7c2-4e47-a656-10a7922ec3e2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.826279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.826966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.827256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.827536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.828284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:44.828656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:44.830252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:44.839443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 241/490] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:44.840180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.846746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a15354cd-4a1a-4cfb-8158-21cb7d624aaf None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.855944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:44.885159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.886303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.886808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.887286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:44.891389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:44.898542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:44.900822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:44.920412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:44.921290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:44.922128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:44.922128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:44.922128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:44.922128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:44.924378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 250/491] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:44.925415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:44.952945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cca22f8b-2050-40ab-bd26-dfa75168095d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:44.989802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:44.990620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:44.990938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:44.991219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.002903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:45.011687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.012603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 242/492] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.013789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.038014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ac6e52-ace5-4736-835d-54cf4b374e00 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.075208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.076396 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.076879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.077352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.089061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.091288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:45.110445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.111390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.112319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:45.112319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:45.112319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:45.112319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:45.114549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 251/493] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.115648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.127796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2bf790-70f4-4e47-91b4-b0246624a00a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.129943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.140510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.171979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.172658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.173063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.173350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.174201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:45.174513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.175998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.194048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 243/494] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:45.194048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.229639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68b18b25-184d-4592-918f-e5a6896e0aff None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.265702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.266371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.266652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.266957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.267700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:45.268004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.269377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8d85bdd-f968-40ef-adac-f728505a41f7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.277613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 252/495] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:45.278952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7660f73-53d0-4697-89a3-3171ec4aa363 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.279238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7660f73-53d0-4697-89a3-3171ec4aa363 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.279517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7660f73-53d0-4697-89a3-3171ec4aa363 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.282890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 253/496] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:45.282890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7660f73-53d0-4697-89a3-3171ec4aa363 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.300783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7776face-9ecb-4396-b449-bad83e5947d2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.337087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.337755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.338043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.338327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.339078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:45.339386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.340772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.357166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 244/497] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:45.357166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.366479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.384266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.405350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7660f73-53d0-4697-89a3-3171ec4aa363 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.446589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.447433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.447691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.447972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.455695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.457145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.464750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b9607c2c-3325-47f4-a34f-502827fb051c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-394084eb-0f4d-4966-9e40-765f3a8eea93 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:45.473813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.474980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:45.475906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 254/498] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/users/nova => generated 82 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.476559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.476559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.476775 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.495281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:45.496675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:45.504839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.505418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.505706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.505979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.517506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.519002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:45.536663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.537711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 245/499] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.538757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.573934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 255/500] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:45 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:45.574811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.623688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.644778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.647386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f334e739-bdb8-498b-933a-9fcd77f3c58d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.681660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.682382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.682671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.682984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] PATH_INFO: `/v3/users/glance` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.683364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148b710b-f720-49b6-ad14-2c5ba0a070ce None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.690776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.692250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:45.708649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.709742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 246/501] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/users/glance => generated 84 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.710954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.720486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.721164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.721448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.721734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.722567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:45.722906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.724361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef922652-998a-4d5e-92c0-189c5a168649 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.761277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 256/502] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:45.762557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884221-f6aa-4445-8ad5-cee35c90b104 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.762877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884221-f6aa-4445-8ad5-cee35c90b104 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.763500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3e884221-f6aa-4445-8ad5-cee35c90b104 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.764559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 257/503] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:45.765082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e884221-f6aa-4445-8ad5-cee35c90b104 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.813624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc66d031-3545-464c-aeed-a350cf6fb26c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.847380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.848040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.848327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.848607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.849327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:45.849638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:45.851043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:45.858730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.864001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 247/504] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:45.864791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.870188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3e884221-f6aa-4445-8ad5-cee35c90b104 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:45.890034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:45.904434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:45.905234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:45.905518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:45.905794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:45.916854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:45.924618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:45.925660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:45.926462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 258/505] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/roles/service => generated 85 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:45.927181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:45.967812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ef8bd3c-e032-42d4-9da3-1af239fdb829 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.002340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.003033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.003323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.003625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.004441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:46.004752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.006191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff9f8a0-47a7-43e6-9a9d-7a5abb1e30b9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.025413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68e4976c-e8d2-47c5-bf56-8964ddcebbfa None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.043196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 248/506] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:46.044445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.044733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.045010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.060412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.061318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.061602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.061878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.061978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:46.063115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:46.112658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.113491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.129777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 249/507] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:46 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:46.130633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.135013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.136500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.235913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 259/508] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:45 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:46.236666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.246339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25b05d8-b46d-47cd-aaf9-e7350b422b91 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.280803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.281587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.281871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.282144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.289041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.291571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:46.306433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:46.307470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:46.308336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 250/509] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/users/cinder => generated 84 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:46.308918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.317677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.338618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7957f88-d57a-4132-9e66-8a7fc32d6644 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.359619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.374088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.374755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.375067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.375345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.376086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:46.376392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.377780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.385573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 260/510] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:46.386353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.419228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80c819f6-086a-488d-b0c8-ccaf9f40f112 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.454286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.454967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.455254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.455533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.490494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97e12c53-b492-4ed6-8b5e-23066681d647 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.521617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.523103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.523207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.524036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.524811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.525089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.535604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.537044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.538730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:46.554735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:46.555703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:46.556749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:46.556749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:46.556749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.556749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:46.556749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 261/511] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:46.557087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.570025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.624755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 251/512] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 317 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:46.625905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.659776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10dab9a7-6558-45bd-b210-5d6edfcf78f9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.692156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.692949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.693230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.693506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.705175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.706638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:46.724018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:46.724958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:46.725797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:46.725797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:46.725797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.725797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:46.725797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 262/513] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:46.726066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdf598e0-fd77-442f-a066-c55627a320bf None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.726136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.760247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.760833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.761114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.761383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.762204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:46.762510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.763966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.771671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.781165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 252/514] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:46.781984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.785812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.827702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab92ba42-8964-4990-9958-8c86ebf7e091 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.861136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.861928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.862212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.862493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.873755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.875264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.883154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5cff06b-e2d3-4e91-bf51-fb41453081c7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:46.892884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:46.893763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:46.894607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:46.894607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:46.894607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:46.894607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:46.894607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 263/515] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:46.894921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.919395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:46.920080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:46.920370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:46.920649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:46.921366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:46.921679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:46.923067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:46.934725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 253/516] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:46.936138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:46.994332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.997350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:46.998264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cd60ddd-106e-445a-8ac9-b3fdf7121004 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.032275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.032988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.033266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.033537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.035508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32778d74-bcb2-42e6-ac0a-4a1a9529f1cb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.044553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.046007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:47.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:47.064366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:47.065206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:47.065206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:47.065206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:47.065206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:47.065206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 264/517] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:47.065477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.074091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.074763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.075082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.075365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.076092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:47.076410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.077780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.090100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 254/518] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:47.091090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.170820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d0d2d80-3a24-401b-ab7a-77e17eebf78e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.189511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b76e06b2-4627-458f-9749-4803a93a222d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.203970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.204620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.204904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.205184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.213682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.224127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.224914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.225607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.225912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.226194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.226938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:47.227336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.228699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.239619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 255/519] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:47.240474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.274512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.276033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.339421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da6eefd5-a192-429c-bbef-93e506ab819d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.371679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.372359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.372482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 265/520] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:47.372630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.372908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.373232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.373608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.375081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.394865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 256/521] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:47:47 2022] POST /identity/v3/users => generated 247 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:47:47.400400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.435522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.453893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.500355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98f63e9-5b7d-4b08-9988-c03a76f9f0d4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.519869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3acd6f3a-a72e-495a-988f-8ecfa1305ef9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.540375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.541057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.541339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.541901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.542653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:47.542989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.544380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.557626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 266/522] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:47.558452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.559595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.560462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.561014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.561297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.569139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.572040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:47.590272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:47.591210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:47.592044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 257/523] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:47.592829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.690966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bddcc54-2a8e-4b82-9614-af848c2b0837 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.703347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.710350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.714621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f08bcb8-8426-41ce-96f7-b0b015db66c0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.733064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.733867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.734146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.734414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.743742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.745199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.757601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.759671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.759960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.760237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:47.760345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:47.761262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:47.762143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 267/524] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/users/neutron => generated 85 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:47.762692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.776186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.777640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:47.796340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:47.797232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:47.798090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:47.798090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:47.798090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:47.798090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:47.798090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 258/525] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:47.798457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.879443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774fcf79-c1e5-459c-88d1-e4565718faee None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.908917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6621ab14-f218-4595-893e-7d6e3e6c2339 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:47.916383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.917196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.917481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.917757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.924844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.926307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:47.941866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:47.942728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:47.943616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:47.948406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 268/526] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:47.949246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.949774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:47.950439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:47.950724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:47.951027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:47.951816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:47.952120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:47.953542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:47.972859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 259/527] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:47.973685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:47.986262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:47.990342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.065596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-391cdc78-2a03-45ea-a9cb-7e6bea6ee801 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.083278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d1e0b1-68a8-433f-8afa-7511ff83e191 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.103786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.105120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.105870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.106296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.107816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.108609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.110666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.117779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.118534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.118849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.119155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.119859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.120169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.121556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.134262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 260/528] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.134921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 269/529] 10.222.0.18 () {56 vars in 1139 bytes} [Sun Jul 24 00:47:47 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.135118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.136077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.218963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.235374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.248996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b0305ee-01d6-43bd-89c8-39ddb32ed6ce None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.254703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41bb56e1-1c99-4100-a213-334e181d973d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.287517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.288169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.288452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.288726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.289641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.289956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.290472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.291159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.291429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.291537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.291707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.292480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.292789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.294465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.307328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 270/530] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.307328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.320161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 261/531] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.321956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.433956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01116002-f58f-44de-bc1e-0006bacda7c0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.451119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47a8473a-1f6a-4e63-9e50-aec2dc7e372f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.473196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.473993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.474271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.474544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.482755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.484484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.491454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.498240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.498945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.499229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.499509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:48.503928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:48.504766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:48.505602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:48.508075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 271/532] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/projects/service => generated 88 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:48.510125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.528510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.586354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.588101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.638250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ddba46-1cb2-4223-8e93-9dc1ebf084e8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.676443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.677026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.677307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.677590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.678428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=e85523ac61714842a523a22a7bc2888b) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.679060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.680925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.694795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 272/533] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/role_assignments?user.id=e85523ac61714842a523a22a7bc2888b&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.694795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.709691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 262/534] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 389 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:48.715425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.794272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.800438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:48.823596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea680f5-7e76-455f-906c-c70f1cf62c0e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.873763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.874444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.874728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.875034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.875838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:48.876150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.877658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:48.910186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eb6b6a7-3b7a-4e5a-bef7-8f35da9de841 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:48.916026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 273/535] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 228 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:48.921786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:48.957388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:48.958124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:48.958412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:48.958692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:48.984060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:48.985520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:49.027488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:49.028413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:49.029239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:49.029239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:49.029239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:49.029239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:49.037152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 263/536] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 322 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:49.038052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:49.126461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-767182d0-16c1-4d29-8bb6-63935c2de28e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.166892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:49.167562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.167847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.168126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.169093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=5cba3eae172d459bb5c29b712945691c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:49.169517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:49.171172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:49.174227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:49.179249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 274/537] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:48 2022] GET /identity/v3/role_assignments?user.id=5cba3eae172d459bb5c29b712945691c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 258 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:49.220454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:49.249310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1916ba87-17d0-4de9-9f3a-fdc743d471bc None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.304276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:49.304961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.306115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.306410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.307177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:49.307492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:49.308958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:49.329713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 264/538] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:49.335515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:49.456200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e22d8258-65ba-46d5-afce-994f87ae22cd None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.494180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:49.494942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.495230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.495508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.502968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:49.504454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-2098884d-a98e-4202-943f-cb6929d6ecac None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:49.519610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:49.520481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:49.521304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 275/539] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:49 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:49.526647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:49.535490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ec8fe-9a76-4674-80fe-b4ad6c753929 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.535780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ec8fe-9a76-4674-80fe-b4ad6c753929 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.536067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c8ec8fe-9a76-4674-80fe-b4ad6c753929 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.537770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 276/540] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:49 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:49.543814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.544103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.544382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.562695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:49.563920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:49.646226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:49.668982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 277/541] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:49 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:49.703859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9330ab4d-62af-419d-aa3c-5251c3120a77 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.708389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:49.773726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:49.774963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.775244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.775518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.776305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:49.776613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:49.778038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:49.807112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 265/542] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:49 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 281 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:49.811834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:49.823794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:49.892907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99de9957-de06-43bd-9efc-97d873fb5ade None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.954336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1a9e5a-a3f3-4c39-b35b-4c3651d3e6b1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:49.958119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:49.958946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:49.959227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:49.959494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:49.984112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:49.991953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:49.992930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:49.993749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 278/543] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:49 2022] GET /identity/v3/roles/service => generated 85 bytes in 285 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:50.003914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:50.014139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:50.014811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.015100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.015375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.016202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=e8144687e0b546108a7a5eb55ac9b452) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:50.016510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:50.017931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:50.030892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 266/544] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:49 2022] GET /identity/v3/role_assignments?user.id=e8144687e0b546108a7a5eb55ac9b452&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:50.131293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:50.210538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81dce29c-df14-46a3-b095-81f3028052fc None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:50.257372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:50.258060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.258899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.259179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.259939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:50.260247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:50.261648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:50.275554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 279/545] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:50 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 272 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:50.281198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:50.411150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:50.461199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c01e4e64-d04a-4053-b551-b439c10655a1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:50.512775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:50.513579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.513865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.514141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.534549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:50.535981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-504a6d09-e24a-46e9-b8df-a19edf3032c6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:50.559008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:50.559899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:50.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:50.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:50.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:50.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:50.561010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 267/546] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 280 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:50.566242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:50.752069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2fb443-347e-4cd0-b633-d85118a93ad6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:50.769908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074be265-97d6-4df6-9013-27aaea6ed240 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.770221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074be265-97d6-4df6-9013-27aaea6ed240 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.770483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074be265-97d6-4df6-9013-27aaea6ed240 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.772177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 268/547] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:47:50.772714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:50.786377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.786644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.786882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.798800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:50.799517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:50.799805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:50.800087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:50.800838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:50.801152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:50.802581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:50.821948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 280/548] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 251 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:50.823217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:50.831790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:50.833002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:50.966577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 269/549] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:50 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:50.967418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-817c7328-77b7-4ba1-8073-588563a8d286 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:50.990170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.032173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.033306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.033814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.034288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] PATH_INFO: `/v3/users/swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.046854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.049211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:51.074867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:51.076344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:47:51.077430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:51.084657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 281/550] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:50 2022] GET /identity/v3/users/swift => generated 83 bytes in 257 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:51.086076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.108204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.123791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.193585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a08d12e-d7be-4098-aca2-67d994c0249d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.208332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f42ccdf-7610-468e-920f-ef862be02aaf None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.245195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.245398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.245944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.246227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.246504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.247150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.247387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:51.247697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.248047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.248530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.249150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c15f0867-6e81-4c01-9d59-4538c2a13956 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:51.265118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:51.274462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 282/551] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:51.274462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102a51d-592c-4d40-aaa3-441435fa607b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.274462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102a51d-592c-4d40-aaa3-441435fa607b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.274462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0102a51d-592c-4d40-aaa3-441435fa607b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.278140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 283/552] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:51 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:51.279002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102a51d-592c-4d40-aaa3-441435fa607b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-13fa044e-4c0b-4c82-8ebe-2a30e9d1f335 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:51.281299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:51.282797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:51.283938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:51.287943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 270/553] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:50 2022] GET /identity/v3/roles/service => generated 85 bytes in 298 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:51.289482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.289927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.290380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.316520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:51.317663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:51.399017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 271/554] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:51 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:51.399933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.410497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.440627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0102a51d-592c-4d40-aaa3-441435fa607b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.447034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.485125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.485790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.486078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.486353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.539567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c438f746-8342-4000-8186-26d7bec610c2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.563202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.564747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:51.594256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.595978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.596275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.596564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.598760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:51.599654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.603110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:51.622870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 272/555] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 219 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:51.623792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.682962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.698266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 284/556] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:51.699175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.737867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f9d8424-eb54-4113-a8ec-eb32449faa30 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.739676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.786979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.787793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.788066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.788339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.801831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:51.810579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:51.813042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:51.815118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:51.822850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 273/557] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/roles/service => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:51.822850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.835990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b984ba70-a77a-4a8a-819a-f6558bd2b805 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.881320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.882034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.882321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.882599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.895506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.897102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:51.918383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:51.919973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 285/558] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:51.920850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:51.939536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56a39262-6dff-4ae6-b015-45ff4a6775a4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:51.971721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:51.981393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:51.982211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:51.982514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:51.982813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:51.996348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:51.998434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-6fb5bb0f-a879-4323-aa53-6c820b42f077 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:52.019897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:52.021390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:52.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:52.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:52.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:52.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:52.022390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 274/559] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 204 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:52.022993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86578f8-d3c2-4fc8-9542-6a75b0f7706c None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.022993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86578f8-d3c2-4fc8-9542-6a75b0f7706c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.023261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b86578f8-d3c2-4fc8-9542-6a75b0f7706c None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.024801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 275/560] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:52.027151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86578f8-d3c2-4fc8-9542-6a75b0f7706c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.045194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.047633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3de8d9a-c5d2-412d-93c0-174311f0894b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.098703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.099413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.099855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.100134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.101457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.101774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.103761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.114784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 286/561] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:51 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.115746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.147474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b86578f8-d3c2-4fc8-9542-6a75b0f7706c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.186725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.187738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.188231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.188694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.189627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.190126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.191537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22847ae0-3e38-48c0-b14f-f81d9b99606e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.208075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 276/562] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.208075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.208075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.208075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.224757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:52.225943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:52.226499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a602ef-afd8-4674-afe7-8f02bbb89ff4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.229816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.265046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.265916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.266298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.266688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.267738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.268194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.269584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.287025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 287/563] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.287025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.313119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 277/564] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:52.316451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.327128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.400616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d473f5e-4aff-4e94-81bd-1c8912d8a16b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.435155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.436215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.436489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.436758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.439719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2cff52c-e118-4d43-baac-04302ece5e5a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.447270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.448724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:52.466937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:52.468033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 288/565] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:52.468902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.478952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.479754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.480041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.480319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.486953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.487521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.488987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:52.504028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:52.505045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 278/566] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/users/nova => generated 82 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:52.505999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.572803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.575075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3dcd66-6d7b-40ba-9fe9-0ca0b61570a3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.609206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.609993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.610271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.610542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.612982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f99b6de6-af29-4224-8b06-c92f00f39f28 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.617286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.618724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c97a4978-ced5-4d5a-9942-7bdbda041d69 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:52.632290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:52.633773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:52.634661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 289/567] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/projects/service => generated 88 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:52.635208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a9ee76-d055-47df-9a1a-b1fe49618604 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.635693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a9ee76-d055-47df-9a1a-b1fe49618604 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.635693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a9ee76-d055-47df-9a1a-b1fe49618604 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.640749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 290/568] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:52.640749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a9ee76-d055-47df-9a1a-b1fe49618604 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.647950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.648787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.649575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.649853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.660103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:52.668684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:52.669728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:52.670568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 279/569] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:52.671302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.720479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.744165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a9ee76-d055-47df-9a1a-b1fe49618604 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.768572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c7af1d9-bfba-433c-9d23-5071d2615f11 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.777587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.778251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.778536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.778837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.779556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.779865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.781238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.791710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.794191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 291/570] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.795025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.806320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.807015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.807296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.807569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.808393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.808698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.810104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c692d586-6e62-42f7-a9aa-5e443b495e29 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.827433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 280/571] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.828616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.828908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.829183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.845194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:52.846363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:52.900558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51198082-5d2f-49af-a3d4-5a3aac01e86b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:52.913087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 281/572] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:52 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:52.913854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:52.952033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:52.952694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:52.952982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:52.953259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:52.954034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:52.954341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:52.955798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:52.958337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:52.968258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 292/573] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:52.968892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.016532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40c429bc-3bf6-4764-9c4d-21f2e088ad1e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.029202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.050577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.051287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.051596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.051903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.052649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.052960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.054339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.062900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 282/574] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.063691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.067066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b004744-59b6-4282-95c9-0acf96c8e628 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.099637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.100422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.100698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.100968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] PATH_INFO: `/v3/users/glance` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.109032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.110483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:53.124954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.126110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 293/575] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:52 2022] GET /identity/v3/users/glance => generated 84 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.127369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.167345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ae8b0-070c-41c9-9783-a5201d0649a7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.184722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.201256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.202048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.202334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.202618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.213604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.215916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.229879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac213e3e-8937-4568-95fe-e0e2c90d7ab8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:53.234955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.235987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 283/576] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.241685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.265209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.269643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.270438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.270710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.271007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.280787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:53.288521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.290018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:53.291044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 294/577] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/roles/service => generated 85 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.292347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.337915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-024579e7-5366-4404-b21b-de53138c35ec None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.370197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.370869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.371151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.371427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.372240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=d1387a119eb04d0a9efe0164de9a8c3c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.372548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.373969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.379318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 284/578] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/role_assignments?user.id=d1387a119eb04d0a9efe0164de9a8c3c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.380173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.391440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d10d090-e715-4a3d-9d18-4ff650233ca4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.420408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.439080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.439904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.440188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.440465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.453590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.455144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:53.474586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.475589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.476463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:53.476463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:53.476463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.476463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:53.481184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 295/579] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.482139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.495218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.508101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde47da0-be68-484e-b637-82e5b803842d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.551736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.552472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.552775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.553036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.553927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.554181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.556509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.578689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 285/580] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.578689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.606332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa254c23-3440-4095-a549-f3846ccd1be8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.641543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.642209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.642496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.642799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.643528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.643925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.645314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.657887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 296/581] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.658705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.688087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.693464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce0a8b0a-23fc-4fea-b038-a0381be8b1eb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.732306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.733031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.733322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.733602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.734480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.734813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.736316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.750099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 286/582] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.750801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.758320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ac484c7-a856-484e-8455-fc7cb52fe493 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.763740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.793802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.794468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.794749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.795048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.795769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:53.796072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.797435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.813845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 297/583] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:53.813845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.868882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bab712fc-2d45-4310-907c-551e4a50ee21 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.903438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.904486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.904782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.905060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.910442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:53.917520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.919010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.925865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e83fb520-3d3f-442e-a7b0-77f903d7f98a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:53.936401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.937288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.938222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:53.938222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:53.938222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:53.938222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:53.938222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 287/584] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.938505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:53.967995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:53.969060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:53.969628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:53.970203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:53.977771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:53.979272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:53.994897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:53.995822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:53.996721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 298/585] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/projects/service => generated 88 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:53.997233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.007701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.054029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9e0b07-f31e-4673-bfa1-aa0ad35af6ee None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.095089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.096181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.096645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.097065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.103834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14bd5b0b-e3a9-430e-8b13-104e1b567e10 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.108765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.110357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:54.128463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:54.129417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:54.130338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:54.130338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:54.130338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:54.130338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:54.136003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 288/586] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:54.136854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.142932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.143744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.144026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.144301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.152249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.153714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:54.168472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:54.169421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:54.170349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 299/587] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:53 2022] GET /identity/v3/users/cinder => generated 84 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:54.171643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.182730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.248586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ddf3fe-2ff8-4acb-b254-adea5c95c6c0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.262722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.281623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7e834ad-ca84-47ca-b5ec-3dcfb0f9a7c3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.286449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.287321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.287757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.288214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.289536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:54.290002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.291901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.307290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 289/588] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:54.308104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.319257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.321141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.321582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.322121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.323470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:54.323956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.326125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.346310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 300/589] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:54.348473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.423694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dc5e16a-106a-4c06-8096-81e98487c598 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.429332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.461519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.462189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.462484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.462753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.463502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:54.463814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.465192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.466077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc47c5f-de4a-4f9c-8aae-ea72e5e253f7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.482392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 290/590] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:54.482392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.509143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.509822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.510168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.510528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.511463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:54.511851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.513348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.520127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.536308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 301/591] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:54.536308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.599853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae5c6cc-fd81-464d-8796-1e354441045c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.635320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.636351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.636797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.637080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.642054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5595dc06-605a-492e-ac86-c80c56ca9621 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.644322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.645803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:54.660709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:54.661602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:54.662493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:54.666966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 291/592] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/projects/service => generated 88 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:54.667814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.669798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.680524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.681323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.681612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.681882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/e06beecc102046d8861a0db046c60528` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.698520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=865a61591b7d45ca90431a7683e96a61, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.700005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.722797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 302/593] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:54 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:54.722797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.798630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8228b631-aae4-45c0-b81d-5574f1656bbd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.807151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:54.844436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.845519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.845809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.846093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.854180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b7c76fd-fafa-412a-a000-f5e2a1aac1ad None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:54.858251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.859775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:54.880839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:54.881826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:47:54.882753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:54.887259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 292/594] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/users/neutron => generated 85 bytes in 220 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:54.888292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.901736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:54.902654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:54.903045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:54.903393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:54.915944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:54.917551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:54.938337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:54.939306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:54.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:54.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:54.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:54.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:54.940197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 303/595] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 227 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:54.941313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:54.971252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.005093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-619443c6-22ac-41ba-ba63-cd6e386f511c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.041659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.042333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.042618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.042921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.043706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:55.044015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.045443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.058488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 293/596] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:55.059252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e0ddd6-70cb-40e9-96ef-9deebc35600a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.059382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.075568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.101306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.102445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.102981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.103457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.104565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:55.105183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.106854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.127622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 304/597] 10.222.0.18 () {56 vars in 1139 bytes} [Sun Jul 24 00:47:54 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:55.128943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.166389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-712b73db-ac42-4e89-a2a0-1602a2b54467 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.203078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.203748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.204088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.204369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.205122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:55.205442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.206831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.223097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 294/598] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:55.223988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.231412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.250183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a04243f-bdf0-44b9-8e87-31145c9b5650 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.292813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.293876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.294160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.294435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/e85523ac61714842a523a22a7bc2888b/roles/e06beecc102046d8861a0db046c60528` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.310065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.311684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=e85523ac61714842a523a22a7bc2888b, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.313286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.329453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 305/599] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:55 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/e85523ac61714842a523a22a7bc2888b/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:55.330683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.341682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4748cfa-b055-4530-a845-ef39a698ae5c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.379649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.380765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.381226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.381657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.394574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.399359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d2cbd2c0-4fb6-4c8c-870d-5bc50fecd2b0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:55.423705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:55.425320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:55.426291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:55.426291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:55.426291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:55.426291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:55.428678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 295/600] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:55.430892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f39b6-e01c-46db-996a-5ed196bfe411 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.431330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f39b6-e01c-46db-996a-5ed196bfe411 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.431755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c7f39b6-e01c-46db-996a-5ed196bfe411 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.433226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 296/601] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:55.434912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f39b6-e01c-46db-996a-5ed196bfe411 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.453031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88a74b0c-417f-45d9-8cb3-5e896cc1f1a1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.495196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.496081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.496366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.496642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.500208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.507834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.510363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d2263b77-b44d-4b89-8b26-852a81380b84 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:55.526380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:55.527386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:55.528291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 306/602] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:55.529490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.530797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.530797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.558057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:55.559665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:55.560523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c7f39b6-e01c-46db-996a-5ed196bfe411 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.596195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.607341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.608014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.608323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.609469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.610210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:55.610525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.611932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.627631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 297/603] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:55.632610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.657020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 307/604] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:55 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:55.658267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.755333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3099a735-46c6-4f27-95b4-d3aca7bd8db3 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.779481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1426ab3-5909-492d-9b5e-d3a306124329 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.792955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.793621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.793907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.794437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.795725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:55.796162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.797995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.808463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.816692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 298/605] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:55.817762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.824427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:55.825238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:55.825519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:55.825792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:55.833604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:55.835128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:55.851837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:55.853905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 308/606] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:55.855424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:55.880501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:55.953235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7201eab8-d226-42ac-bc5d-41a99c1646bc None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.969356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3be6646-b7ef-403f-a201-6ed13f0f6a8f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:55.998386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.000081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.000555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.001116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.003833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.004902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.005183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.005456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/5cba3eae172d459bb5c29b712945691c/roles/e06beecc102046d8861a0db046c60528` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.016712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:56.026531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=5cba3eae172d459bb5c29b712945691c, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:56.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:56.028005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:56.028837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:56.029554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.037230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 299/607] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:55 2022] GET /identity/v3/roles/service => generated 85 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:56.037230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.051345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 309/608] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:55 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/5cba3eae172d459bb5c29b712945691c/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:56.052610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.103520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:56.187508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29435339-91bb-4f75-8043-6fcc6ce4167c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.191750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dde2dd5d-59c9-4c2f-b947-5dabf664895f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.203344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:56.230514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.231473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.232295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.232725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.233665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:56.234119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.236148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.251666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 310/609] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:56.253311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.271183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.271838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.272514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.272514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.273262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:56.273603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.275072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.309325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 300/610] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 274 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:56.315312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.364838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:56.369431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b8dc616-8636-4d2e-9523-ba954c843932 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.409024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.410189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.411050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.414858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.427427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.429183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:56.454820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:56.455795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:56.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:56.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:56.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:56.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:56.456656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 311/611] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:56.456936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f74e68-1b70-46d1-b9a8-2a8da54348e5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.464524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.525457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.526352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.526631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.526931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/e8144687e0b546108a7a5eb55ac9b452/roles/e06beecc102046d8861a0db046c60528` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.559543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=e8144687e0b546108a7a5eb55ac9b452, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.562203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.583826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 301/612] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:56 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/e8144687e0b546108a7a5eb55ac9b452/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 269 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:47:56.649649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b826696-1b10-487a-bce6-bd09b055ceda None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.697183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.697852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.698135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.698408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.699159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:56.699463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.701093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.717455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 312/613] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:56.723411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.743376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:56.843441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fad0dd88-75a4-47ba-a2a1-634b19805d4c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:56.890274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:56.891125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:56.891408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:56.891681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:56.903271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:56.904643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4e2a456d-9c3f-4dc4-b644-60b249b4997e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:56.931446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:56.932462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:56.933809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 302/614] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/users/swift => generated 83 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:56.940515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:56.954132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.050649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d2011e-4a07-477b-888b-9468afeb68ca None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.098687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.099393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.099666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.099933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.100762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:57.101064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:57.102527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:57.106279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd156b4-8af3-4b0f-a83a-37aa54297146 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.106566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd156b4-8af3-4b0f-a83a-37aa54297146 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.106867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7cd156b4-8af3-4b0f-a83a-37aa54297146 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.108495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 303/615] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:57.113620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.113911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.114194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.133902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 313/616] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:47:56 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 194 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:57.134101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:57.135705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:57.139245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.212542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 304/617] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:47:57.238191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.255146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.347168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.353654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df3a0a73-1953-4e9c-b2b1-f3c915606a48 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.388257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.389045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.389322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.389594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.404294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:57.406073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:57.430038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:57.431233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:57.432104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:57.432104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:57.432104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:57.432104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:57.434143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 314/618] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 295 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:57.440168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.459327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc2e4e7a-d21e-487b-9486-d302903ba129 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.531248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.532050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.532344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.532623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.550482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:57.558637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:57.559695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:57.560944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:57.568723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 305/619] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 331 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:57.576634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.620931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54df2e8c-0b54-4e86-b015-84e482f3de76 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.643111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.658569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.659378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.659672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.659954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.660687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:57.661000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:57.662623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bba1c4f-50b2-483b-9167-321e70469522 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:57.666481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.677853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 315/620] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 238 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:47:57.679292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d784a51-89dc-4662-82d1-84ae39325a5d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.679594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d784a51-89dc-4662-82d1-84ae39325a5d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.680531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d784a51-89dc-4662-82d1-84ae39325a5d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.682020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 316/621] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:57.684745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d784a51-89dc-4662-82d1-84ae39325a5d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.714453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f153ce4d-938c-4864-8bae-5e635610f534 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.750237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.751012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.751302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.751583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.752450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:57.752764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:57.754202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570ec6b7-e3c2-41dc-b654-bcbe7774cd13 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:57.763359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 306/622] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:57.764614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.764913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.765194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.782655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:57.783879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:57.815508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d784a51-89dc-4662-82d1-84ae39325a5d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:57.853964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:57.855126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:57.855323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:57.855605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:57.863212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:57.864702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:57.874974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 307/623] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:57 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:57.875757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:47:57.890436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:57.891362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:47:57.892237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:57.895733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 317/624] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/projects/service => generated 88 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:47:57.896473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:57.944086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:57.958679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.012071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31722b63-0113-4e5f-981c-6a8fbe803c6e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.013994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81457104-056f-4c92-85a6-22ca9020a5f6 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.049936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.050984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.051552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.051884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.057645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.058992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.059428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.059852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.064427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:58.071756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:58.073290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:58.074405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:58.075479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:58.076456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.081328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 318/625] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/roles/service => generated 85 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:58.082255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:58.098516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:58.099596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:58.100508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:58.100508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:58.100508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:58.100508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:58.104031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 308/626] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:57 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:58.105255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.207236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.211700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-954a6ffe-8f3e-438f-aee2-aa0089c8130f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.225960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e11ce0d-47ee-4f8f-90f7-a75b3f71b004 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.240067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.254469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.255425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.255941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.256365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.257398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:58.257897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.259656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.268704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.269524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.270333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.270725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.271882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:58.272313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.274319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.275119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 319/627] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:58.276224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.289881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 309/628] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:58.289881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.401742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87eff8ba-9622-4a53-96eb-b242e89568b4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.404393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3fe666e-97e3-42ce-9977-d25e21fb04d8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.439770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.441098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.441661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.442142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/d1387a119eb04d0a9efe0164de9a8c3c/roles/e06beecc102046d8861a0db046c60528` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.447738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.448661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.449146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.449624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.450618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:58.451183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.452880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.458670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=d1387a119eb04d0a9efe0164de9a8c3c, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.460156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.466148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 320/629] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:58.467419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.480663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 310/630] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:47:58 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/d1387a119eb04d0a9efe0164de9a8c3c/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 195 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:47:58.482024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.498323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.515586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.599106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e1e0c8-9f54-4eec-90b4-fe65194caf7d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.615550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9e66da3-52e9-4617-b889-bd06addcf5ca None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.639142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.639954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.640244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.640523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.649531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.651058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.662920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.663728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.664006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.664498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-6f4ca556-f4b6-4fe2-9903-b3c663c002d6 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:58.667662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:58.668604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:47:58.669555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:58.673769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 311/631] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/users/nova => generated 82 bytes in 188 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:58.675318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488ae1e2-7782-4637-8fdf-18882f28d454 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.675603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488ae1e2-7782-4637-8fdf-18882f28d454 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.675886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-488ae1e2-7782-4637-8fdf-18882f28d454 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.678335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.678567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 312/632] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:58.678567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488ae1e2-7782-4637-8fdf-18882f28d454 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.678878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-04e0b5c6-1098-4155-8df8-30b6d4daaf29 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:58.707533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:58.708472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:58.709350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:58.709350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:58.709350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:58.709350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:58.712684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 321/633] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 245 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:58.714438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.714438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.714628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.742331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:58.743631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:58.786520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.819707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-488ae1e2-7782-4637-8fdf-18882f28d454 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.830792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 322/634] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:58 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:58.830792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.831749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:58.879746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:58.880818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:58.881373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:58.881855 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:58.884141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:58.884673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:58.886894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:58.911363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 313/635] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 233 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:58.914887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:58.956300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-733d40b2-ae23-4bb8-9df3-9a8281663eea None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:58.999465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.000321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.000608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.000889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.001662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:59.001975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.003658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.022843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 323/636] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:59.024155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.051462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f107321c-d600-437f-aa31-86d9e1b6dbd8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.098926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.100640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.100928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.101204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.107075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.115571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.122356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:47:59.134915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:59.136580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:47:59.137487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:59.145291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 314/637] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:58 2022] GET /identity/v3/roles/service => generated 85 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:59.145291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.159889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a2d220f-a237-44e3-b83a-644b430e83ee None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.203565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.204137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.204437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.204716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.263739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5f84d77-8673-4610-b9b1-91dd8e5e81d0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.277183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.278902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42163a52-e4a5-4ab9-8dd8-b1449eeffc62 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.299650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.300621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.301345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.301797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] PATH_INFO: `/v3/users/glance` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.310999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] RBAC: Authorizing `identity:get_user(user_id=glance)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.312555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] Could not find user: glance.: keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:47:59.327081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:59.328038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance. Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:59.328960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 315/638] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/users/glance => generated 84 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:59.330484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.347466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 324/639] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 365 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e4346-b9c9-495f-8097-41faa25d5eb9 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e4346-b9c9-495f-8097-41faa25d5eb9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-594e4346-b9c9-495f-8097-41faa25d5eb9 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 325/640] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:47:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:47:59.392726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e4346-b9c9-495f-8097-41faa25d5eb9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.408399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.457736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a600f5f-da4f-4dee-89ee-7b099376570c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.495577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.496389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.496679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.496987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.497734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:59.498051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.499454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.507713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 316/641] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:59.508471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.515110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-594e4346-b9c9-495f-8097-41faa25d5eb9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.550173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.550872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.551161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.551688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.552268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] RBAC: Adding query filter params (domain_id=default, name=glance) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:47:59.552582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.554027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d4ff243a-c879-4ef5-b084-4e857c77f2ff None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.576913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 326/642] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/users?domain_id=default&name=glance => generated 372 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:47:59.576913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.576913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.576913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.594056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:47:59.595640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:47:59.611176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.618639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165318b3-5bd9-410b-8737-c51244eb5b05 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.649079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.662414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.663250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.663540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.663942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.675369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.676916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.694805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 327/643] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:47:59 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:47:59.694805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:59.707431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:59.708694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:59.709752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:59.709752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:59.709752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:59.709752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:47:59.715130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 317/644] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:59.715130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.801168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ae7c6c2-bfe8-4ff6-9b4a-f300028df396 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.816041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3fe326a-3307-49ac-85fa-d89d23ef21a7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:47:59.838290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.839129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.839415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.839694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.850098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.851613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.852454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:47:59.853105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:47:59.853504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:47:59.853783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:47:59.870087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:47:59.871048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:47:59.872096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:47:59.872096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:47:59.872096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:47:59.872096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:47:59.872096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 328/645] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:47:59.872386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:47:59.875201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.904059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:47:59.932431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:47:59.934254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:47:59.980137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b864cc07-6a91-44b6-9440-4ad918486244 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.017083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.017884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.018170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.018458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.028674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:00.036656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:00.037614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:00.038456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 329/646] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/roles/service => generated 85 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:00.039716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.052286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 318/647] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:47:59 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:00.052286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.108850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.130536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.152406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ade539b3-7829-44c0-b964-12f0f9d37063 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.158243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5011a6a1-b363-4274-a512-bbe32b90582d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.191063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.191896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.192282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.192713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.193476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.193884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.195146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.195581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.196021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.196113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.196381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.197101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.197584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.199039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.208307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 330/648] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.215148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 319/649] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.215148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.215406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.319075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45fbc039-56e2-4ad8-8407-49d8160388c9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.321914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a741aa1-be0a-4825-8307-b5ec97514b97 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.347432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.356174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.356979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.357254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.357532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.359357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.359692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.360024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.360308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.360589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.361341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.361655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.363283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.370297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.371764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.372169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 331/650] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.372968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:00.395583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:00.401872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 320/651] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:00.403015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.501392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5da42d1-c358-432b-92ee-0a7ab509a368 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.511266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14a3480-4dc0-4067-b89d-9736746708d9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.541221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.542025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.542310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.542584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.547641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.548442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.548728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.549016 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.549647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.551123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.557673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.559147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:00.566107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:00.567002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 332/652] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:00.572913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:00.574231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:00.575220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:00.576147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 321/653] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/users/cinder => generated 84 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:00.577678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.625666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.626380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.680561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-007798bc-17b0-407b-8cf3-c8b84453d87a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.681392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16c126ce-f8e2-4a35-b93f-94bad109312f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.714247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.714912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.715197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.715479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.716186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.716286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.716490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.716996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.717279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.717552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.717840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.727657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.729091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.729580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 322/654] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.730217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:00.748222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:00.749168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:00.750034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:00.750034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:00.750034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:00.750034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:00.750034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 333/655] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:00.750460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.828349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e4e427e-5fab-419c-82fd-5ed88c0ddfa8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.835253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.848245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:00.854721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df834be-a45c-42cb-ba4f-e7a94f287cf1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:00.862692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.863403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.863691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.863967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.864741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.865054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.866459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f9d5a96-ed7c-4faf-8492-19450ec33ca4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.877729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 323/656] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.878972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d34f47-f8fc-49ed-ac9d-a2be8210579e None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.879263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d34f47-f8fc-49ed-ac9d-a2be8210579e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.879547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15d34f47-f8fc-49ed-ac9d-a2be8210579e None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.880933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 324/657] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:00 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:00.881528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d34f47-f8fc-49ed-ac9d-a2be8210579e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.891004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:00.891684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:00.891968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:00.892246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:00.893084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] RBAC: Adding query filter params (domain_id=default, name=cinder) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:00.893393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:00.894845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:00.912480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 334/658] 10.222.0.18 () {56 vars in 1137 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/users?domain_id=default&name=cinder => generated 372 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:00.913268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:00.990133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15d34f47-f8fc-49ed-ac9d-a2be8210579e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.017983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e559cdb5-ea06-41f5-96ec-0d1eebcfc6f2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.023642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.024398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.024680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.024955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.031658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.033034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:01.048744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:01.049858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 325/659] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/projects/service => generated 88 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:01.051266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.052935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.053597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.053874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.054151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.055557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:01.055849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.057217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18f659a6-af3c-4937-bec5-c9d7cd9dfc72 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.069062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 335/660] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:01.070306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.070609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.070914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.083163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.087322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:01.088781 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:01.098251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.156136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b89654f9-50e5-4bfb-a82a-f46ec23a992b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.157229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 336/661] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:01 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:01.158044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.204732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.205389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.205673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.205949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.206864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:01.207222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.208655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.215207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 326/662] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:01.215980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.262664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c8700c5-ecae-4ffa-86d1-d359a7b1779e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.304861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.305802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.306091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.306371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.321958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.324888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.333897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c534bf8b-3185-4ead-9a95-524941b7c66e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.375782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.376823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.377349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.377696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.378590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:01.378841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.380309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.391195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.393190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.393314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 327/663] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:01.394024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.506125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 337/664] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 348 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:01.507435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.519364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea52ea97-4d5e-4eb5-9780-d40c00a6a499 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.554975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.555946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.556304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.556664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] PATH_INFO: `/v3/users/neutron` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.560274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.565017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] RBAC: Authorizing `identity:get_user(user_id=neutron)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.566463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] Could not find user: neutron.: keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:01.581657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:01.582646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: neutron. Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 328/665] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/users/neutron => generated 85 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:01.583640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.598267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.624896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65e0c3-b14f-4246-aeed-3d9045cbcb09 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.662242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.663163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.663456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.663727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.674858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:01.683086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:01.684025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:01.684852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 338/666] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:01.685569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.701630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f88a557a-dcc4-4e74-9746-6c0637b29499 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.738802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.739732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.740184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.740716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.741740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=e85523ac61714842a523a22a7bc2888b) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:01.742240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.744078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.750923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 329/667] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/role_assignments?user.id=e85523ac61714842a523a22a7bc2888b&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:01.751881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.807538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1817efe-7eb0-4246-81ab-46fe4fdfb572 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.812169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.856081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:01.862314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.864128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.865141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.865910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.883623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.885870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.886920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09e30815-3696-4edf-9802-538af6802ab1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:01.911897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:01.913050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:01.914752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:01.914752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:01.914752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:01.914752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:01.917772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 339/668] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/domains/Default => generated 87 bytes in 231 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:01.917772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:01.940823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:01.942049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:01.942497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:01.943018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:01.944255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] RBAC: Adding query filter params (domain_id=default, name=neutron) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:01.944736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:01.946762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:01.969194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 330/669] 10.222.0.18 () {56 vars in 1139 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/users?domain_id=default&name=neutron => generated 374 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:01.969986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.050997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bafc9f5-999a-47a6-b29c-d49a4155705d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.086975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.087747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.088110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.088447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.089492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:02.089626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.091108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.096746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a01e29ea-ea56-45e7-95d3-7d8b17bb65e8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.100329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 340/670] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:02.100891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.126455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.139032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.151257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.151946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.152302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.152654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.153459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:02.153847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.155329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.168876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 331/671] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:01 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:02.169918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.244530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2fd185e0-ac04-4739-b50c-e1f677f442a9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.283452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4c5426d-66b3-4032-81ec-3dd6c9f3d664 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.291863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.292734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.293644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.294072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.321752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.322763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.323177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.323597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.335221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.337220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:02.356280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:02.357439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:02.358614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:02.358614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:02.358614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:02.358614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:02.360294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 332/672] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 191 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:02.361137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.396386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.399250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.413315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.425315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.472901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d26e1e0f-4e8e-4dbf-a2b3-faf2b3acb629 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.509859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.510782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.511145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.511508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.518896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.519306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 341/673] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:02.520343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.520574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:02.536319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:02.537463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:02.538598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:02.540405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 333/674] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/projects/service => generated 88 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:02.541368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.632222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2325b8a-b337-426a-884e-62a4aed0a59b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.648710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0acbc06d-7e7c-47e8-ac03-1b885505465e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.651224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.653109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.673403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.674384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.674873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.675356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.676290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:02.676802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.678359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.686750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.687575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.687865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.688145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.697786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 342/675] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:02.698922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.699734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.701197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:02.720293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:02.721238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:02.722122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:02.722122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:02.722122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:02.722122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:02.722122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 334/676] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:02.722940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.807920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3d21c55-dc02-47d0-94df-695040714f89 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.832783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f035249-efd0-4a77-a800-50debf2dd1f5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:02.844199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.845200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.845740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.846273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.847291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:02.847814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.849449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.862612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 343/677] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:02.863822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.870054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:02.871102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:02.871391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:02.871674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:02.879635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:02.881156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:02.897283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:02.898318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:02.899391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:02.903696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 335/678] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/users/swift => generated 83 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:02.904544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:02.912632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.924078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:02.980413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fcab42ed-58d9-484e-bce1-ce3eabf8b7c1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.015190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d06f37-56ed-481d-a5b5-3ab0467cf94e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.018033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.019044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.019600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.020075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.021001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:03.021518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.023129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.036045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 344/679] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:03.037193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.060294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.061343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.061538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.061818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.062708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=5cba3eae172d459bb5c29b712945691c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:03.063149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.064672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.072301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 336/680] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:02 2022] GET /identity/v3/role_assignments?user.id=5cba3eae172d459bb5c29b712945691c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:03.073081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.149287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.155003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bec6b4-7de0-4f91-b906-111f59f53654 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.167092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.190340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-714c3321-3038-44b3-b4c7-14cc75675337 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.194404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.196007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.196515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.196987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.198057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:03.198599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.200372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.220692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 345/681] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:03.228156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.242716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.243834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.244121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.244399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.259404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.262027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:03.284393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:03.285384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:03.286259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 337/682] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:03.293713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.386958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a0920d1-ff9d-47fa-b09d-b024c5a5d64d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.414959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-530f146c-a4ac-43ce-876d-154b21516fb0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.438906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.440279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.440984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.441419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.454831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.455717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.456096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.456481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.457362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:03.457773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.459340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-776773e9-996d-443d-aef8-3df4bf55059c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.472157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 338/683] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:03.472465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.478236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e37f348-94c6-4f59-8acb-824c894b8f47 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.478502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.478626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e37f348-94c6-4f59-8acb-824c894b8f47 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.478944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e37f348-94c6-4f59-8acb-824c894b8f47 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.480637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 339/684] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:03 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:03.482700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e37f348-94c6-4f59-8acb-824c894b8f47 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.547306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.549861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.592563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e37f348-94c6-4f59-8acb-824c894b8f47 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.639358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.640432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.640927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.641413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.642989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=e8144687e0b546108a7a5eb55ac9b452) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:03.643544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.646023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0278b01a-f327-49de-a319-1e837c7dbb15 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.663422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 340/685] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/role_assignments?user.id=e8144687e0b546108a7a5eb55ac9b452&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:03.663422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.663422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.663422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.687475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 346/686] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 358 bytes (2 switches on core 0) Jul 24 00:48:03.693819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.697081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:03.699714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:03.789928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.832980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:03.848034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 341/687] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:03 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:03.865114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c70eca89-882e-4241-9669-c515f6eca44e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:03.876583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:03.914400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:03.916256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:03.916541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:03.916814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:03.928147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:03.930268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:03.954251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:03.955214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:03.956123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:03.956123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:03.956123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:03.956123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:03.961240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 347/688] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 263 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:03.962486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.012950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-608bfc75-e01b-473c-9c96-8dba604d59e5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.058805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.058805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.058805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.058805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.068472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:04.077958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:04.078961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:04.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 342/689] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/roles/admin => generated 83 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:04.086397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.145849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b87649c-439e-4955-b35a-c1b8062ccec9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.152080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.158562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.194412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.195605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.195896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.196173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.197200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.197511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.199462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04517364-10f5-453a-9905-24edee61fa9b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.207670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31c9655b-309e-412b-a434-2a2bfaabf042 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.216411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 348/690] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 254 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:04.218604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7996cc-6893-4eaa-9b43-38c00b7da656 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.219563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7996cc-6893-4eaa-9b43-38c00b7da656 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.220337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f7996cc-6893-4eaa-9b43-38c00b7da656 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.222430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 349/691] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:04 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:04.223788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7996cc-6893-4eaa-9b43-38c00b7da656 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.248512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.249139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.249421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.249697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.250561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.250839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.252251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8b87c68-7a72-4379-b3a5-e6cf47e4081d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.261631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 343/692] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:04.266221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.266221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.266221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.284352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:04.285350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:04.371211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 344/693] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:04 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:04.372085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f7996cc-6893-4eaa-9b43-38c00b7da656 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.372202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.416374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.417290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.417676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.418340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.428611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.431090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.444979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:04.451374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:04.452451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:04.453572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:04.455766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 350/694] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/projects/service => generated 88 bytes in 232 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:04.456767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.488561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.522617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd5258d0-f3ce-423d-b723-59ff59aefea0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.566981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.567416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2c2254a-de23-47e9-b5f7-9543c147f284 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.568041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.568425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.568803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.582331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.585064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.609277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.610395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.610946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:04.611077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:04.612400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:04.613283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:04.613283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:04.613283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:04.613283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:04.614018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.614904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 345/695] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 243 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:04.615921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.628888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.630681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:04.650632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:04.651610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:04.652525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:04.652525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:04.652525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:04.652525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:04.659357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 351/696] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:04.660748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.738119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.744266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07dd497f-c6d2-42a1-9c89-496bc7c0a771 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.746174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:04.768403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e1e0906-8e34-4d2e-aaa2-e93a7b91090f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.787286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.787954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.788242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.788522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.789612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.789929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.791397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.807469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 346/697] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:04.808743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.809423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.810213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.810502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.811065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.811799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.812107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.813776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.827620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 352/698] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:04.828833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:04.943004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e63fe521-c0a7-414f-885e-ac443604a675 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.953875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-339f7684-5a64-45f2-a55f-8186bf5c2875 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:04.979139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.979812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.980099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.980367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.981096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.981409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.982820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:04.994058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:04.995278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:04.995784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:04.996336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:04.997370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=d1387a119eb04d0a9efe0164de9a8c3c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:04.997887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:04.999575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.000976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 347/699] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:05.001902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.008469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 353/700] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:04 2022] GET /identity/v3/role_assignments?user.id=d1387a119eb04d0a9efe0164de9a8c3c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:05.010014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.014563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.034819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.138249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7221743-3013-4c76-a194-0a1ffe47a679 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:05.176953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5527d0ed-4598-4508-8918-12163af02392 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:05.184382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:05.185069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.185352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.185873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.186587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:05.188537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.218824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 354/701] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:48:05 2022] POST /identity/v3/users => generated 254 bytes in 206 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:05.222992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:05.223810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.224092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.224419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] PATH_INFO: `/v3/users/nova` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.234166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:05.235734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:05.253152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:05.254125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:05.255123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 348/702] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/users/nova => generated 82 bytes in 253 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:05.260277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.332525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.378953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95c2db10-e6f8-466b-8e6c-d77a858f0003 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:05.419585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:05.420280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.420546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.420815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.421651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:05.421953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:05.423427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a9c1863-2021-420a-b605-f36f0a2455fd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.442950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 355/703] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:05.448559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.538539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.541352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2962a7a-8ac1-453a-9054-338d1d67be74 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.541824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2962a7a-8ac1-453a-9054-338d1d67be74 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.542115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2962a7a-8ac1-453a-9054-338d1d67be74 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.543706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 356/704] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:05.550641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.550966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.551243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.569370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff81e6a-8982-4018-b633-56e9629df1bb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:05.575219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:05.576594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:05.605352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:05.606009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.606292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.606570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.654158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 357/705] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:05 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:48:05.677886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.678073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:05.679402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.732767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.788703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a666fd14-62dd-4e24-8fbd-ef00ee2b1aed None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:05.816798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:05.838263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:05.839620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:05.840109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:05.840581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:05.841438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:05.843110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:05.850630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 349/706] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/auth/tokens => generated 799 bytes in 402 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:05.859646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:05.860609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:05.861452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:05.862246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.864519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 358/707] 10.222.0.18 () {56 vars in 1129 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:05.871393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:05.989095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42c6a910-e59e-4186-961a-5af9998036bb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.000624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-097c4560-e7fd-4a51-ace9-517d389ddb99 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.007218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.029408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.030495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.031011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.031568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.032583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:06.033106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.034697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.036934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.037773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.038022 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.039709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.045175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 359/708] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:06.049421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.051193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.051704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:06.071488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:06.072529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:06.073453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:06.073453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:06.073453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:06.073453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:06.073453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 350/709] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:05 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:06.077988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.112549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.178783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd7bf066-87c8-4680-ae50-cc25867488b3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.231527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.232298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.232508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.232790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.233516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:06.233827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.234421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85058118-8d1d-49ce-93d3-656102d659ed None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.235262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc7279bd-4d5f-4a1b-9eea-b29c17ded18d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.249704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 360/710] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/services?type=block-storage => generated 130 bytes in 199 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:06.272639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.273950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.274240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.274515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.275300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:06.275606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.277008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.281279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433717d-3887-4778-a7f9-d232fa1a5b38 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.281572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433717d-3887-4778-a7f9-d232fa1a5b38 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.281856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3433717d-3887-4778-a7f9-d232fa1a5b38 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.283615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 361/711] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:06.289224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.289532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.289886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.290730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 351/712] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 213 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:06.296413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.343958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:06.345207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:06.403567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.414875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4556b7df-4fd4-436d-a546-c4948008f211 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.453465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.454265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.454552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.454869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.464691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.466137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.471113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:06.484915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:06.485914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:06.486925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 352/713] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:06.492428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.523466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 362/714] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:06 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 235 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:06.545266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.678232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0a57ae5-7265-42d0-a71b-4e122e11a262 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.722806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51019320-0f32-48cc-a73a-1335759f5866 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:06.764217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.765211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.765694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.766193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.767181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:06.767672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.769835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.783545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 353/715] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 292 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:06.792374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.794594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:06.797570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:06.798038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:06.799102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] PATH_INFO: `/v3/services/network` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:06.799981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:06.807820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:06.821816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:06.822205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:06.823285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:06.824231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:06.827788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 363/716] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/services/network => generated 88 bytes in 283 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:06.833201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:06.911284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:06.954610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71055b24-46be-42ad-9dbf-29e4e938f89a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:07.020667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17cfa377-6260-4c9f-882f-b8c86262f4fc None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:07.049546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:07.050119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:07.050404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:07.050679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:07.053836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:07.054160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:07.055665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0848fa3b-c56c-476c-8d8a-056b72c41629 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:07.067690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:07.068362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:07.068644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:07.069321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:07.070696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] RBAC: Adding query filter params (name=network) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:07.071197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:07.072749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:07.077574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 364/717] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/services?name=network => generated 124 bytes in 245 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:07.077745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 354/718] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:06 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 286 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:07.083303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:07.260651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:07.265143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b78de8b0-3a63-4d5c-b5b1-b6456f69f0a4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:07.323361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:07.324307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:07.324591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:07.324863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:07.325588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] RBAC: Adding query filter params (type=network) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:07.325890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:07.327299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34102ed2-fc72-4864-8bc6-6d528ad67b3d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:07.334393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 365/719] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:07 2022] GET /identity/v3/services?type=network => generated 124 bytes in 252 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:07.937769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043f551-06a6-4666-bafe-b2fc2d2e9a79 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:07.938789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043f551-06a6-4666-bafe-b2fc2d2e9a79 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:07.938789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6043f551-06a6-4666-bafe-b2fc2d2e9a79 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:07.941462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 355/720] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:07 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:07.950051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54573a-6476-4f04-b828-cb3401e44638 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:07.950531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54573a-6476-4f04-b828-cb3401e44638 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:07.951083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc54573a-6476-4f04-b828-cb3401e44638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:07.994895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bc54573a-6476-4f04-b828-cb3401e44638 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:07.996734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bc54573a-6476-4f04-b828-cb3401e44638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:08.093662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 366/721] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:07 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:08.156056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043f551-06a6-4666-bafe-b2fc2d2e9a79 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:08.327724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:08.389229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6043f551-06a6-4666-bafe-b2fc2d2e9a79 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:08.420948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423436f7-765d-4b7b-bf4f-34b83df57884 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:08.421447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423436f7-765d-4b7b-bf4f-34b83df57884 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:08.421940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-423436f7-765d-4b7b-bf4f-34b83df57884 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:08.425014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 367/722] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:08 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:08.434094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:08.434599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:08.438792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:08.453234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:08.454568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:08.455095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:08.455588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:08.484865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:08.486894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:08.494899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:08.519924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:08.520902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:08.521748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:08.523386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 356/723] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:08 2022] GET /identity/v3/roles/admin => generated 83 bytes in 368 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:08.531405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:08.681188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 368/724] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:08 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 248 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:48:08.687492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d61d60d8-9f31-4a1a-b5ec-f4e9d67bdddb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:08.722177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:08.722877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:08.723168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:08.723450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:08.724198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:08.724506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:08.725924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:08.727936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:08.742135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 357/725] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:08 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:08.743926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:08.750322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:08.792896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.004059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9322c46-40e7-4341-91b4-e1ef2084e815 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.011834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd291f13-57b2-4da7-a788-48c3d74d1baa None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.075298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.076155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.076975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.077256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.077532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.078627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.079004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.079326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.101022 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.102508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.103808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:09.118085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:09.119076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:09.119922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 369/726] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:48:08 2022] GET /identity/v3/roles/service => generated 85 bytes in 377 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:09.129581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:09.134978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:09.136131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:09.138522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:09.138522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:09.138522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:09.138522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:09.138522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 358/727] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:08 2022] GET /identity/v3/domains/Default => generated 87 bytes in 387 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:09.144299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.229843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.257254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.295007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d25aa4ea-2359-4c98-af9e-07a3a204a584 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.302878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86070557-f4dd-4d74-b9c4-d8c20759371d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.339374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.340051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.340344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.340623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.341346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:09.341655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.343051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.356128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 359/728] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:09.363739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.371035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.371712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.372192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.372476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.373232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:09.373539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.374951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.393737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 370/729] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 264 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:09.399172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.533370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2da9743a-4209-4dec-ba46-93c0cb23bd73 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.542568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-974c3f8a-2dbd-4c08-9946-36fc25ec5ba7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.546760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.580098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.581254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.581607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.581717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.581902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] PATH_INFO: `/v3/users/swift` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.582420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.582700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.582999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.592149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.594516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.595206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.596765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.600157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4808d5a0-079c-4d70-966a-906e5d705d5b None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:09.614210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:09.615223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:09.616171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 360/730] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/users/swift => generated 83 bytes in 252 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:09.617745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ab8ef-2684-4a29-b793-8a5212a50635 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.617745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ab8ef-2684-4a29-b793-8a5212a50635 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.617745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d8ab8ef-2684-4a29-b793-8a5212a50635 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-96f3d823-f0e4-452e-97c8-239fa826b208 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:09.622793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:09.624079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:09.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:09.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:09.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:09.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:09.625082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 371/732] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/domains/Default => generated 87 bytes in 221 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:09.625417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 361/732] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:09.625500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.625500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.625500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.626933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8ab8ef-2684-4a29-b793-8a5212a50635 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.646211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:09.647854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:09.734934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d8ab8ef-2684-4a29-b793-8a5212a50635 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.746876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 372/733] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:09 2022] POST /identity/v3/auth/tokens => generated 799 bytes in 119 msecs (HTTP/1.1 201) 6 headers in 363 bytes (2 switches on core 0) Jul 24 00:48:09.747770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.793970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.794934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.795678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.796324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.797159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:09.797570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.800295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.817058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 362/734] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:09.818070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.850358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.872074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fdb00c5-9248-4eab-8b53-a19393d52a44 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.899554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:09.909825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.910496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.910806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.911083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.911909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:09.912214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.914465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.933759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6779d751-1f0f-4a5e-8e0e-ec29e622853a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:09.934919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 373/735] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 187 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:09.936244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:09.972901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:09.973596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.973890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.974120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.974993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:09.976934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e7eaf35-d153-4937-944d-6c25b703c730 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 363/736] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:09 2022] POST /identity/v3/services => generated 251 bytes in 174 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc80b5c-f3a5-49c7-ac8a-71c7cccdb5e8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc80b5c-f3a5-49c7-ac8a-71c7cccdb5e8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fc80b5c-f3a5-49c7-ac8a-71c7cccdb5e8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 364/737] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:09.998809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc80b5c-f3a5-49c7-ac8a-71c7cccdb5e8 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.055339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dae3fb24-d433-4b99-867b-4cbeb89ee68d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.103374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.104311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.104593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.104867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.113322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.114901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.121074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-e408667d-bc43-40a3-b842-43763dbf85bf None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:10.131800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:10.134736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 374/738] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0199f5-5e07-4e5b-b7ac-28aa3762cdf5 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.137877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0199f5-5e07-4e5b-b7ac-28aa3762cdf5 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.140722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee0199f5-5e07-4e5b-b7ac-28aa3762cdf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.142352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fc80b5c-f3a5-49c7-ac8a-71c7cccdb5e8 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.153657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ee0199f5-5e07-4e5b-b7ac-28aa3762cdf5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:10.154914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ee0199f5-5e07-4e5b-b7ac-28aa3762cdf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:10.182990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.183959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.184251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.184527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.195335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.232253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 375/739] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:10 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:10.233594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15700e54-a009-4412-aa43-e5efd4c4b5c2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.233888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15700e54-a009-4412-aa43-e5efd4c4b5c2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.234168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15700e54-a009-4412-aa43-e5efd4c4b5c2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.235729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 376/740] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:10.236423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15700e54-a009-4412-aa43-e5efd4c4b5c2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.280101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.281692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-749e92be-8e7f-46b5-bbc3-428e841bc374 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.350285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15700e54-a009-4412-aa43-e5efd4c4b5c2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.381558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.388972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.389641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.389929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.390210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.391085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:10.391390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.395493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.404617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 365/741] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:09 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:10.406046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.406341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.406625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.414427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 377/742] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:10.414427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.429877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:10.431111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:10.441982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.503283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 366/743] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:10 2022] POST /identity/v3/auth/tokens => generated 903 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 363 bytes (1 switches on core 0) Jul 24 00:48:10.504133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.524937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a69259-9faa-4c42-a151-0553fff68c33 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.560199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.561312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.561647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.561935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.573090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:10.580914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:10.581853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:10.582692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 378/744] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:10.583614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.626394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-deb21e63-24a6-4695-8cea-9769965af248 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.644883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.667474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.668277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.668560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.668837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.678218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.681140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.682714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.693300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c30ad1fc-bfc4-422a-812d-c88ac3dd2364 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:10.701363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:10.702427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:10.703562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:10.703562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:10.703562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:10.703562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:10.707227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 367/745] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/domains/Default => generated 87 bytes in 203 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:10.707935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.732998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.733945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.734229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.734509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.803899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.805412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.815646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96e78849-81d2-4c9f-b75e-7e93fcab9a84 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:10.852969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:10.853638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:10.853922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:10.854200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:10.854950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:10.856451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:10.871432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 368/746] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:10 2022] POST /identity/v3/services => generated 240 bytes in 164 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:10.872232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:10.907912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.923374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:10.934554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 379/747] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/auth/tokens => generated 903 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 358 bytes (1 switches on core 0) Jul 24 00:48:10.936576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.030480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41d3b00-8dc5-410d-a3b0-6ee26212309d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.078440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b906ff-e6af-4aba-a842-d7d22f027d12 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.078644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.079299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.079587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.079868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.080621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.081602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.082309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.092905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 369/748] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 221 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:11.094092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.123398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.124080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.124421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.124713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.125553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.125967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.127481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.142683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 380/749] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:11.143609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.200586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.214894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efd7fb6d-f845-4be3-82ae-28c23d1000d7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.218902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.260614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.261576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.261859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.262135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.276400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.277851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.278216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccd0c582-8edd-4548-9051-0721900bbeb6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:11.300100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:11.301224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:11.302201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:11.302201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:11.302201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:11.302201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:11.306610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 370/750] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:11.307705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.321105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.322340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.322869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.323421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.335017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.336788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:11.357373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:11.358300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:11.359259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:11.359259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:11.359259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:11.359259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:11.365500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 381/751] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/domains/Default => generated 87 bytes in 218 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:11.366477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.417148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c21909b-96e2-4da3-bac4-2ac01f7c0bef None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.452758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.453516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.453753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.454027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.461129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.462594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.465162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.475195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:11.478898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:11.480200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:11.481315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:11.482649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2206e755-3f8d-4ee9-869a-a563854f8b12 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.484763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 371/752] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:11.486285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.519242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.520231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.520699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.521212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.522159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.522647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.524455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.536835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 382/753] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:11.537904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.590045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4e6868b-11da-4763-8a91-8485c57c37a8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.627290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.628059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.628348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.628621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.629366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.629827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.631146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.646086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe61907-82ce-4808-a10b-254fb2c1713c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.649184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 372/754] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:11.650475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.684967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.686350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.686870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.687407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.688474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.689006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.690721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.704609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 383/755] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:11.705769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.720767 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.729697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.770101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91d67f01-ed0d-4c03-8782-42e0b9dab13f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.807116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.808143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.808650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.809063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.812569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7368e1f2-0999-4b44-b4cf-342b765f2f66 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.816980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.818589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:11.834814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:11.835899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:11.837046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:11.839246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 373/756] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:11.840399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.851519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.852532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.853036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.853291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.861428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:11.863054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:11.878138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:11.879280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:11.880433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:11.886743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 384/757] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:11.887773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:11.954499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d28dc680-2839-461f-877a-21fb0534f95d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.973318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.978306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:11.995147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cb9e7be-5e46-44ea-93ed-395e121b7345 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:11.996296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:11.997216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:11.997675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:11.998177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:11.999182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=d1387a119eb04d0a9efe0164de9a8c3c) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:11.999640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.001203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.014935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 374/758] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/role_assignments?user.id=d1387a119eb04d0a9efe0164de9a8c3c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:12.016217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.044313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.044993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.045279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.045556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.046367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:12.046678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.048202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.077613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 385/759] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:11 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:12.083589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.187935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fac80493-0fe2-43d8-8402-c374c87ef827 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.230086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.230679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.230998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-243f6b5e-75d4-49e4-b683-a238ebe551df None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.231122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.231299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.232113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:12.232540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.234087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cacf5f92-2323-457e-8313-d401e56d247d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.253246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 375/760] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:12.259640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c470d36-f834-46be-9331-10997b37eee7 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.259936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c470d36-f834-46be-9331-10997b37eee7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.260218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c470d36-f834-46be-9331-10997b37eee7 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.262003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 376/761] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:12 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:12.262733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c470d36-f834-46be-9331-10997b37eee7 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.269935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.270687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.271044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.271325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.272287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=f2b854f126714f82bfa98223c38e7376) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:12.272604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.274754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a869ebf9-4717-4959-b5f5-04b64280e31a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.286810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 386/762] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/role_assignments?user.id=f2b854f126714f82bfa98223c38e7376&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:12.286810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97354009-f62b-45bd-9f80-655617f80181 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.286810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97354009-f62b-45bd-9f80-655617f80181 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.286810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97354009-f62b-45bd-9f80-655617f80181 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.294806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:12.323957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-97354009-f62b-45bd-9f80-655617f80181 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:12.325187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-97354009-f62b-45bd-9f80-655617f80181 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:12.325842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:12.427827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c470d36-f834-46be-9331-10997b37eee7 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.469923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.470845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.471135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.471416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.484066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.485508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.487545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 387/763] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:12 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 206 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:12.521049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:12.522203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:12.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:12.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:12.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:12.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:12.523291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 377/764] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/domains/Default => generated 87 bytes in 260 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:12.529251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97354009-f62b-45bd-9f80-655617f80181 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.530106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.668691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:12.677485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6b3e2d-8d69-4681-9a14-d2f9a4e7dd74 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.710805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97354009-f62b-45bd-9f80-655617f80181 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.716186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.717196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.717639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.718096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.718934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.720552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:12.736201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:12.737363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:12.738329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:12.740749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 378/765] 10.222.0.18 () {56 vars in 1129 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 211 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:12.748278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.785644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.786319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.786603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.786907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.787635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:12.787939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.789856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.799744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:12.808364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 388/766] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 279 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:12.814604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:12.915813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ddc59b0-f3ea-4af7-a884-3d1522f19abd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:12.983074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:12.983897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:12.984947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:12.985406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:12.986459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:12.989397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:12.990996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:12.998077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 379/767] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 250 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:13.003636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.015241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b77e2c-84b9-4e0d-9809-51a70f50cfe7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.045531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.071307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.072266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.072657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.073037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.081888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.083701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-6bf86482-5fa9-42f2-86d3-5ceb8eb8a6c9 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:13.100128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:13.101255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:13.102325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:13.104412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 389/768] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:12 2022] GET /identity/v3/projects/service => generated 88 bytes in 290 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:13.105874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51d2d3-5a0e-45be-b349-320449be7115 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.106308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51d2d3-5a0e-45be-b349-320449be7115 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.106729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb51d2d3-5a0e-45be-b349-320449be7115 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.112567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 390/769] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:13.113527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51d2d3-5a0e-45be-b349-320449be7115 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.135866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a900395f-c2fe-41b7-8600-fc08aa3a5176 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.176243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.177593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.178005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.178640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.179929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.180188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:13.181645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.184344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e03a181a-a34b-4852-b415-86e5a05dd299 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.193485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 380/770] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 190 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:13.195795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.196274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.196973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.219457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:13.221266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:13.235155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb51d2d3-5a0e-45be-b349-320449be7115 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.282027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.282817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.283182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.283538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.284667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:13.284779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.286449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.299720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 391/771] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:13.300810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.323460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 381/772] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:13 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:13.324388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.344593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.442238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b886cc3-21b5-439a-a22c-dd910963c0fb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.456802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.464894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7b6e8f5-6205-42ab-954e-ee7606359ede None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.481852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.482520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.482828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.483105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.484056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] RBAC: Adding query filter params (interface=public, service_id=96d77e7b6b62419e9ee83bfb577eb5e3, region_id=RegionOne) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:13.484365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.486097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.500537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 392/773] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/endpoints?service_id=96d77e7b6b62419e9ee83bfb577eb5e3&interface=public®ion_id=RegionOne => generated 194 bytes in 196 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:13.500537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.507068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.508989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.509430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.509934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.512306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=865a61591b7d45ca90431a7683e96a61, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.514150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.535100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 382/774] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:13 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:13.536347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.612694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22a809e9-d3be-49ee-b2cd-c6fa29f122a7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.652244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.653217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.653570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.653911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] PATH_INFO: `/v3/services/network` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.654643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.656272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:13.670415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:13.671745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:13.672771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:13.675392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 393/775] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/services/network => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:13.687932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:13.702554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.753864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c862b632-37bb-4614-931e-9d8a1fefa483 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.776173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:13.813149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.813749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.814031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.814306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.815574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.818580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.824489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 383/776] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/services => generated 772 bytes in 288 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:13.858277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb4c4da8-4e2b-4785-8f05-37e7269a3444 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:13.900729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:13.901717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:13.902148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:13.902599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:13.903513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] RBAC: Adding query filter params (name=network) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:13.903987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:13.905590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:13.911577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 394/777] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/services?name=network => generated 124 bytes in 224 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:13.918164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.027643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac2260bd-6afa-4d23-968a-1b25b962ca42 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:14.063936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:14.064842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.065276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.065720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.066575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] RBAC: Adding query filter params (type=network) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:14.067058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:14.068586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:14.075271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 384/778] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:13 2022] GET /identity/v3/services?type=network => generated 350 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:14.081520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.191148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416b1d0f-b952-4b3c-88ea-1364f3c34c98 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:14.198701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:14.234005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:14.234980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.235506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.235929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.236870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] RBAC: Adding query filter params (interface=public, service_id=51799b4224974989b583769bf15020e3, region_id=RegionOne) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:14.237330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:14.238894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73b74eb9-4a79-492c-b807-a63067ec7130 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:14.246969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 395/779] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:14 2022] GET /identity/v3/endpoints?service_id=51799b4224974989b583769bf15020e3&interface=public®ion_id=RegionOne => generated 194 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:14.252516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.256689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:14.387886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6de486d-4d08-4ecd-8bd2-6471dea5257d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:14.424248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:14.424927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.425216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.425494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.426174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:14.427585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ffe467-2d46-4d1c-9c2f-f35d52d23246 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:14.433889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 385/780] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:14 2022] GET /identity/v3/services => generated 772 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:14.489185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67274c8b-ed11-42be-8a12-d510e2159405 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.489478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67274c8b-ed11-42be-8a12-d510e2159405 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.489749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67274c8b-ed11-42be-8a12-d510e2159405 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.491498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 396/781] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:14.497401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbc6471-8b3c-4fd1-903e-7234162af33f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.497693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbc6471-8b3c-4fd1-903e-7234162af33f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.497976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdbc6471-8b3c-4fd1-903e-7234162af33f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.524247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-bdbc6471-8b3c-4fd1-903e-7234162af33f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:14.525510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bdbc6471-8b3c-4fd1-903e-7234162af33f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:14.610864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:14.622106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 386/782] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:14.647250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67274c8b-ed11-42be-8a12-d510e2159405 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.676539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578b7d-88a3-4a4b-8a12-1d3c1ceb9ae2 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.676848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578b7d-88a3-4a4b-8a12-1d3c1ceb9ae2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.677101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b9578b7d-88a3-4a4b-8a12-1d3c1ceb9ae2 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.678822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 387/783] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:14.684903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.685141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.685386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.703989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:14.705179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:14.785191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67274c8b-ed11-42be-8a12-d510e2159405 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:14.791105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 388/784] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:14 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:14.810994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.861161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:14.863023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:14.863676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:14.865226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:14.888194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:14.896397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:14.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:14.898345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:14.910971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 397/785] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 261 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:14.916647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:14.967134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:15.011445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb405d7-f035-4a76-bce6-25fd296f10db None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.067213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:15.083125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.083941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.084223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.084498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.097636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:15.107896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:15.108851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:15.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 389/786] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:48:14 2022] GET /identity/v3/roles/service => generated 85 bytes in 300 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:15.114894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.117949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b59ef33-38a3-41d5-8659-ec117c9d406d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.194672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.195394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.195676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.195955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.196708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:15.197015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.198618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.216206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 398/787] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 299 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:15.221330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.250706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d55dfa10-88ee-4c5c-b7c0-f93a1edfa7ed None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.284654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.285371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.285655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.285982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.286724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:15.287069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.289120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.302160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 390/788] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 183 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:15.303522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.385317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:15.414790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7b5da05-9d54-4357-8a9f-43086c60a3a5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.451090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.452121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.452409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.452684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.464880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.466530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:15.485305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:15.486221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:15.487160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:15.487160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:15.487160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:15.487160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:15.487160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 391/789] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:15.499616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.531142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:15.543903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c962c32-f861-4894-b14b-85224e05fb47 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.605530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.606329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.606608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.606907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.622052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.623555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:15.652919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:15.653989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:15.654907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:15.654907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:15.654907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:15.654907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:15.657170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 399/790] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/domains/Default => generated 87 bytes in 435 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:15.661610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.684722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf91729e-c6de-4359-9806-c2f8601dec40 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.688330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:15.738760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.739423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.739712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.740039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.740766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:15.741076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.742486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.760102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 392/791] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 261 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:15.765416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.858155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ead36b62-f580-46cf-8e37-663036e5c1ad None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.927811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:15.928485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:15.928770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:15.929072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:15.929809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:15.930107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:15.931530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:15.959182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bef14099-2b7f-4c53-84b2-2fb9deb5b2d4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:15.961593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 400/792] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 300 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:15.964233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:15.968972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.025260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.026459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.026987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.027460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.041849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.043654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:16.073350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:16.074357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:16.075324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:16.083540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 393/793] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 318 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:16.088078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.133189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.149490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e667c5ad-a2e0-4040-8476-9b0db9d651d9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.194540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.195982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.196255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.196528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] PATH_INFO: `/v3/users/swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.212804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.214472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-eb24a9af-7b88-4c4e-9713-928701fc344e None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:16.230099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:16.231140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:16.232076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 401/794] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:15 2022] GET /identity/v3/users/swift => generated 83 bytes in 268 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:16.233150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b449-c091-406d-bb9c-89cbdadab387 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.233325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b449-c091-406d-bb9c-89cbdadab387 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.233576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b449-c091-406d-bb9c-89cbdadab387 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.238808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 402/795] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:16.238808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306b449-c091-406d-bb9c-89cbdadab387 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.276129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e93a909-109d-4a40-bfb7-731b60436ba0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.307733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.319512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.321056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.321563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.322036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.323138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:16.324124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.326136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02dbb768-abdc-4e9f-b073-fc657416290e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.352010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 394/796] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 260 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:16.352010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.352010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.352010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.364494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306b449-c091-406d-bb9c-89cbdadab387 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.367047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:16.368642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:16.410959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.412096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.412611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.413089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.414449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:16.415003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.417908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.443378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 403/797] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 207 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:16.444882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.454797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 395/798] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:16.454797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.473557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.564302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cc6afb0-becc-4400-9ce0-f36c6e6240c6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.577205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f9494d-6efe-482d-8ae7-af20f63e73fb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.587044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.599331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.600128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.600405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.600674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.611397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.612937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.619293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.620089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.620367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.620637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.631867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:16.632018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:16.632896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:16.640145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:16.640145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:16.640145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:16.640145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:16.640145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 404/799] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:16.640423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.640505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-64e19b29-8c77-45c8-bd21-2d9f13c776e6 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:16.658203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:16.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:16.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:16.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:16.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:16.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:16.660294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 396/800] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/domains/Default => generated 87 bytes in 210 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:16.661773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700160b1-6285-4713-9ffd-eb9b6070e620 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.662125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700160b1-6285-4713-9ffd-eb9b6070e620 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.662466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-700160b1-6285-4713-9ffd-eb9b6070e620 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.664892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 397/801] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:16.665672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700160b1-6285-4713-9ffd-eb9b6070e620 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.736603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.761334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb32861f-9c01-4eab-a93a-fbe821ebf6f6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.781131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-700160b1-6285-4713-9ffd-eb9b6070e620 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.796704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.797505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.797786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.798059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.808841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:16.816879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:16.817852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:16.818984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 405/802] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/roles/admin => generated 83 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:16.820127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.820978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.821644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.821935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.822216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.823240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:16.823555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.824926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28f12c12-9a94-455f-bec9-f5fa85205751 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.837613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 398/803] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:16.838982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0abfe8f-a8ac-46ec-a928-3a22adfb4285 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.839285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0abfe8f-a8ac-46ec-a928-3a22adfb4285 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.839565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0abfe8f-a8ac-46ec-a928-3a22adfb4285 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.854992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:16.856294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c0abfe8f-a8ac-46ec-a928-3a22adfb4285 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:16.857498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c0abfe8f-a8ac-46ec-a928-3a22adfb4285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:16.920973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864f24d7-6299-464a-aabd-639f6d55a180 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:16.923092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 399/804] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:16 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:16.924547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c043cb-f02e-43ad-90ae-1b0b768285b3 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.924849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c043cb-f02e-43ad-90ae-1b0b768285b3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.925133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c043cb-f02e-43ad-90ae-1b0b768285b3 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.926531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 400/805] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:16 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:16.927182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c043cb-f02e-43ad-90ae-1b0b768285b3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.962677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:16.963375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:16.963659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:16.963938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:16.964661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:16.964973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:16.966367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:16.978493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 406/806] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:16.979317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:16.986887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.033766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c043cb-f02e-43ad-90ae-1b0b768285b3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.070838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.071426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.071716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.072003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.072755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.073065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.074322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.074432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-920744a2-c316-47a7-905c-ff8abef8c458 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.080942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3654a7d9-2279-4022-8777-1416b68ee074 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.083154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 401/807] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.084359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.084651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.084934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.102279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:17.103504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:17.117059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.117852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.118131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.118404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.125016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.126456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:17.140404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:17.141511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:17.142714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.146253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 407/808] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:16 2022] GET /identity/v3/projects/service => generated 88 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:17.147039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.172209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 402/809] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:17 2022] POST /identity/v3/auth/tokens => generated 1002 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:17.173373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.217747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.264043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f0beec-d930-4f3c-8a75-0a25830dcfb5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.270485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ce9b8e-473d-4ac0-804a-6f7601127041 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.297160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.297965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.298244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.298523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] PATH_INFO: `/v3/services/block-storage` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.299204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] RBAC: Authorizing `identity:get_service(service_id=block-storage)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.300577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.307206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.307995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.308274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.308545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] Could not find service: block-storage.: keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:17.312810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:17.313926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: block-storage. Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.314951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 408/810] 10.222.0.18 () {56 vars in 1129 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/services/block-storage => generated 94 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:17.315805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.315881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.316772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:17.331682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:17.332567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:17.333485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 403/811] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:17.333976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.338986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.419776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d54fab4-e06e-4e15-8684-6d4055c9c492 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.436017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c06a495a-668d-4810-8cbf-ebd3a6364e45 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.438263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.454337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.455164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.455449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.455723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.466869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.468313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.471098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.471762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.472050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.472329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.473108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.473419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.475286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:17.485139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:17.486000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:17.486935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:17.486935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:17.486935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:17.486935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.486935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 409/812] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:17.487225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.492453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 404/813] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.492453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.558467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.588453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e50b5f9e-f653-46ab-a4a7-d82917ed2e19 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.591621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c33435b-c391-4842-86e0-fe2436668638 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.623014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.623665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.623949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.624229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.624848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.624974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] RBAC: Adding query filter params (name=block-storage) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.625207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.625638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.625923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.626199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] PATH_INFO: `/v3/services/network` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.626557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.626878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] RBAC: Authorizing `identity:get_service(service_id=network)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.628232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.630888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 405/814] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/services?name=block-storage => generated 130 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.631662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] Could not find service: network.: keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:17.641032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:17.642070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: network. Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:17.643319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 410/815] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/services/network => generated 88 bytes in 155 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:17.644124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.660660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.733075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e43ea8-f12c-4097-a316-8a6ff7ed77cd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.742435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c22f9244-8a57-4110-9620-3f690a1d0619 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.765360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.766020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.766317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.766598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.767413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.767729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.769144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.775856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.776516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.776803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.777079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.777788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.778095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.779485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.780929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.781508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 406/816] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.782345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.793021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 411/817] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.793844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.873904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:17.886029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd7b01f2-d175-455c-8d83-f481b5114bf7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.891260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f68b073-8dee-4b1b-92c2-76f307be0f9d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:17.920190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.921080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.921366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.921644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/f2b854f126714f82bfa98223c38e7376/roles/e06beecc102046d8861a0db046c60528` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.925399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:17.926066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:17.926350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:17.926624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:17.927362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] RBAC: Adding query filter params (type=block-storage) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:17.927674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.929110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.934310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 412/818] 10.222.0.18 () {56 vars in 1124 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/services?type=block-storage => generated 367 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:17.934953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:17.941039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=f2b854f126714f82bfa98223c38e7376, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:17.942480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:17.965471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 407/819] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:17 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/f2b854f126714f82bfa98223c38e7376/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:17.965471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.001501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.051200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b0f443-a268-480d-a290-7a65f9990856 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.083223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ce9e172-daa8-4467-9d55-d6627c51e4d5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.089764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.090449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.090737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.091044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.091797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] RBAC: Adding query filter params (name=network) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:18.092102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.093507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.098965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 413/820] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:17 2022] GET /identity/v3/services?name=network => generated 124 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:18.099789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.134357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.135610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.136091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.136542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/d1387a119eb04d0a9efe0164de9a8c3c/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.138861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=d1387a119eb04d0a9efe0164de9a8c3c, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.140485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.149899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.156235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 408/821] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:17 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/d1387a119eb04d0a9efe0164de9a8c3c/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:18.165043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.223283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7857a74a-7056-41e4-8146-4429237b461e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.255114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.269998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.271480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.271990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.281974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.284065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:18.302254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:18.303358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:18.304860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:18.307945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 414/822] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:18 2022] GET /identity/v3/users/nova => generated 82 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:18.309238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.421261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37108560-94df-47b4-ba3c-5246bd87cf16 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.436361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba88dc20-5a83-4e95-90aa-4b05737b2ff0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.462613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.463572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.463853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.464124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.465314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.466967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.472857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.473519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.473862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.474141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.474873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] RBAC: Adding query filter params (type=network) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:18.475184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.477126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.482364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 415/823] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:18 2022] GET /identity/v3/services?type=network => generated 350 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:18.483216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.502999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 409/824] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:18 2022] POST /identity/v3/endpoints => generated 346 bytes in 341 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:18.504527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.539531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.596638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e5d0072-cd3a-424f-83b2-b581597eb0be None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.626348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.641783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.642696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.643025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.643293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.644241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:18.644564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.646025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.667086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 416/825] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:18 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:18.672427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:18.746961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bb5047-ac53-4da3-82d6-a23cfdfb09e1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.791281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.792338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.792856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.793351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:18.794403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:18.796098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:18.820830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 410/826] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:18 2022] POST /identity/v3/endpoints => generated 327 bytes in 317 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:18.853151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:18.857082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00625424-00a8-4987-8f3d-b89462729fb7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:18.911232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:18.912424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:18.912969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:18.914098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:19.016017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:19.017407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:19.120283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:19.128898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 417/827] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:18 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 457 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:19.134052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:19.260689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6bfabb3-38e6-4687-843c-9df7441cb817 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:19.274181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:19.305157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:19.306315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:19.306856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:19.307360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:19.318824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:19.320768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:19.339560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:19.340512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:19.341399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:19.341399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:19.341399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:19.341399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:19.345702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 411/828] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:19 2022] GET /identity/v3/domains/Default => generated 87 bytes in 212 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:19.352866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:19.459569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40714394-845b-4f20-9766-73e0b0b84438 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:19.495965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:19.496631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:19.496910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:19.497185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:19.497900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:19.498200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:19.499601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:19.512665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 418/829] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 160 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:19.517803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:19.622041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a3938bb-e0ce-4dc7-a7a4-2a1f5aedcf0a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:19.656718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:19.657840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:19.658332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:19.658844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:19.666636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:19.668461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:19.683699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:19.684577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:19.685551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:19.688225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 412/830] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:19 2022] GET /identity/v3/projects/service => generated 88 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:19.693597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:19.751976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:19.805782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c79dd17-0a6b-4021-ae2e-d56084963011 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:19.840611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:19.841268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:19.841549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:19.841818 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:19.842591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:19.842926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:19.844364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eba475cd-1205-428f-9715-b87df8351643 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:19.858045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 419/831] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:19 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:19.863664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:19.889410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:19.970749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6248243b-47f5-463d-9c07-3a4f4e2ef7b2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.007966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.008653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.008938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.009220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.010066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:20.010383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:20.011868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6e3cecd-be1c-4b71-85ba-2976915d3085 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:20.018748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 413/832] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:19 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:20.144540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d18550-5117-49d5-af3a-7996616de5d0 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.145062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d18550-5117-49d5-af3a-7996616de5d0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.145302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6d18550-5117-49d5-af3a-7996616de5d0 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.147366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 420/833] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:20 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:20.153876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6055ee2d-eadd-4b67-bfaf-59825407680a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.154179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6055ee2d-eadd-4b67-bfaf-59825407680a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.154452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6055ee2d-eadd-4b67-bfaf-59825407680a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.173783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6055ee2d-eadd-4b67-bfaf-59825407680a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:20.175040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6055ee2d-eadd-4b67-bfaf-59825407680a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:20.250025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 414/834] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:20 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:20.275005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d18550-5117-49d5-af3a-7996616de5d0 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.378888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6d18550-5117-49d5-af3a-7996616de5d0 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.385412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce0d7b-77b6-413e-97ba-bcbcc6e7556d None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.385692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce0d7b-77b6-413e-97ba-bcbcc6e7556d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.385970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1ce0d7b-77b6-413e-97ba-bcbcc6e7556d None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.387861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 415/835] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:20.399157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.399758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.400257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.407747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.423458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:20.424905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:20.429349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.430213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.430503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.430814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.439014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.442638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:20.451848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:20.452820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:48:20.453676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:20.456472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 421/836] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/roles/service => generated 85 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:20.463504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.518814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 416/837] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:20 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:20.531233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.588311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5feaa059-9747-474e-b8f3-c5a6d8ccfa59 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.627623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.628672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.629316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.629826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.630880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:20.631392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:20.633857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:20.639572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68240f77-d2e2-475f-af41-05f725d31e3b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.647111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 422/838] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:20.650232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.676049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.676576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.676857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.677128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.689203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:20.698087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:20.698263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:20.699158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:20.699989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 417/839] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:20.705429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.718993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.775220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6d5b2d3-fe9f-484b-9ae9-97a5cea27c5a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.814306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.815508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.816132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.816715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.833020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:20.835023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:20.841479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-292ff7fe-0c50-4ea2-a3fe-9b0794549498 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:20.855514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:20.856440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:20.857332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:20.857332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:20.857332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:20.857332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:20.863175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 423/840] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:20.865673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.879791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:20.880469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:20.880755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:20.881035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:20.881798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:20.882924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:20.885220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:20.898845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 418/841] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:20.903172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:20.974065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.978158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:20.989607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665175f0-5721-4f1f-9032-5ad89d623a49 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.013272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8899e7d8-3a94-414a-a913-ba3addd7dadb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.027019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.027734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.028019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.028299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.029011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:21.029313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.030669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d8d763d-18bc-4021-b539-a4aec765592c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.047390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 424/842] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:21.047390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ca5ad-5e23-4a8e-bf71-cabc8a7b737c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.047390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ca5ad-5e23-4a8e-bf71-cabc8a7b737c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.047390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b8ca5ad-5e23-4a8e-bf71-cabc8a7b737c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.047390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 425/843] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:21 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:21.050760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8ca5ad-5e23-4a8e-bf71-cabc8a7b737c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.056157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.057185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.057648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.058101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.071198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.072852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4229a87c-1e71-4704-96ae-4329e1614eb0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:21.091636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:21.092563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:21.093454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:21.093454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:21.093454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:21.093454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:21.097498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 419/844] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:20 2022] GET /identity/v3/domains/Default => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:21.098606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.099077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.099325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.115695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:21.116897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:21.163383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b8ca5ad-5e23-4a8e-bf71-cabc8a7b737c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.186707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 420/845] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:21 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:21.187523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.246086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.300355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b653aef6-0180-4a4c-9a26-d4ee88e613d3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.322988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.323799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.324078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.324352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.331421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.332847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.337721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.338388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.338674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.338990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.339708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:21.340015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.341377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:21.348572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:21.349494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:21.350427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 426/846] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 299 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:21.351662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.358056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 421/847] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:21.358056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.366380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.462716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00cbaff9-9217-4dbf-9e71-a55a690cebce None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.479611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d67b1ac-eee9-442c-a84b-f55dd2f5f120 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.490177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.499219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.500026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.500311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.500827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.501232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.502608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:21.515112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:21.516081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:21.522076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:21.522891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.522891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.522891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.522891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.523518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 427/848] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:21.523607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:21.523681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.523759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.523759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.539102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 422/849] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:21.539935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.608823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.637840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e10d821-f5ae-4065-bd7d-a0d1c5e12bb7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.641724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90daec74-f38d-4eda-b955-92df8a7a7e22 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.673362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.674080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.674362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.674731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] PATH_INFO: `/v3/users/swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.675311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.675921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.676207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.676486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.677193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:21.677506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.679529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.683686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] RBAC: Authorizing `identity:get_user(user_id=swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.689063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 423/850] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:21.689063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.689292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] Could not find user: swift.: keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:21.705287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:21.706519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: swift. Jul 24 00:48:21.707700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:21.711086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 428/851] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/users/swift => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:21.711086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.723782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.798918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ffaee3-e484-40ea-8ee9-bc35f0a1d61b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.818973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687e6dc6-4865-495f-9090-412e294ed150 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:21.835572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.836648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.836942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.837226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.858082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:21.858748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:21.859060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:21.859333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:21.860033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:21.860341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.861709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.862950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:21.867587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 429/852] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/services?type=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:21.868692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:21.923984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:21.925837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:21.962339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.035981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bb87cb0-c09b-4a76-96a1-baab6bf5a99d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.055798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 424/853] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:48:22.062368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.081900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.082960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.083451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.083953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.085150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] RBAC: Adding query filter params (domain_id=default, name=swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:22.085982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.087466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.105687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 430/854] 10.222.0.18 () {56 vars in 1135 bytes} [Sun Jul 24 00:48:21 2022] GET /identity/v3/users?domain_id=default&name=swift => generated 370 bytes in 237 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:22.111684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.171631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.192143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb950e33-4d0d-4da1-80ba-49bbd9713616 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.219681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2ebefb0-7eb8-4d8d-95a9-2e4acd489954 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.223784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.229370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.230168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.230450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.230723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.241779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.243272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.256821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.258279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.258567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.258875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-535d244b-0cbf-4314-878a-edebd2e2ef5e None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:22.261688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:22.262546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:22.263469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:22.263469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:22.263469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:22.263469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:22.263469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 425/855] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:22.264148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775afe87-9c4b-401a-ac23-e97812d8e636 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.265079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775afe87-9c4b-401a-ac23-e97812d8e636 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.265079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-775afe87-9c4b-401a-ac23-e97812d8e636 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.266079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 426/856] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:22 2022] GET /identity => generated 270 bytes in 2 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:22.270572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775afe87-9c4b-401a-ac23-e97812d8e636 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.342489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.344017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.387305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-775afe87-9c4b-401a-ac23-e97812d8e636 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.404479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.426278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.427135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.427426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.427707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.428430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:22.429328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.430818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e633158-4273-454c-b5c0-ca3a572f4da2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.443821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 427/857] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:22.445801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.446161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.446511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.465586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:22.466778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:22.469510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 431/858] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/auth/tokens => generated 1151 bytes in 358 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:22.470386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.510068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.542948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 428/859] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:22 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:22.542948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.581630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a21caf8-8eb5-45c8-bafa-02be99febef1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.625139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.626761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.627221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.627653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.638401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.640673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.647271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11e17a8f-c554-4360-aea2-0ea82c87416b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:22.658283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:22.659260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:22.660149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 432/860] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/projects/service => generated 88 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:22.660624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.678930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.689822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.690404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.690685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.691020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.701997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.703405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:22.721202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:22.722109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 429/861] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:22.723489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.750206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.774856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ba6ee98-ffc0-4823-9306-a855a7061613 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.809221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.810116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.810467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.811118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.811903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.813371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.825113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f2acf2-7529-43d2-8b35-2c31ec26f335 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:22.827233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:22.828326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:22.829206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 433/862] 10.222.0.18 () {56 vars in 1132 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:22.830037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.868717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.869676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.869954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.870230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.871144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:22.871454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.872905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.885259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 430/863] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:22.886292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:22.922336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.943539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c71a7897-17d5-47db-96c1-da6be8020875 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.975429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:22.982218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:22.982917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:22.983383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:22.983512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:22.984178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:22.984485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:22.985854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:22.993711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98316b6f-c0a1-40f8-a102-fd14cf55ac6d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:22.999869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 434/864] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:23.001010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:23.035403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.036079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.036360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.036639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.037345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:23.037650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.039100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.044712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 431/865] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:22 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:23.045328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:23.112599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-390d733c-d186-4a7e-a509-5e6314b16ece None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:23.147275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1de7959a-68bf-4ce0-b042-e7b9794096b9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:23.148166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.148910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.149271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.149551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.150374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=f2b854f126714f82bfa98223c38e7376) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:23.150687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.152161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.166802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 435/866] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:23 2022] GET /identity/v3/role_assignments?user.id=f2b854f126714f82bfa98223c38e7376&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:23.166802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:23.171090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:23.201087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.202218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.202972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.203336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.216056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.218240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:23.239361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:23.240331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:23.241233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:23.241655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 432/867] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:23 2022] GET /identity/v3/projects/service => generated 88 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:23.248274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:23.263274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:23.290110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a01de3-91ce-453a-a4e4-0bf9178eceb8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:23.330878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.331776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.332059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.332332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.333481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:23.334860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.337587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.342962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 436/868] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:23 2022] GET /identity/v3/services?type=compute_legacy => generated 131 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:23.388394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94fae8ac-76fa-4369-b926-58b5ed569d38 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:23.431799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.432488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.432776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.433051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.433857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:23.434172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.435923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b11ad7b-9478-4db3-b4b4-9296a072de7a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.451746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 433/869] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:23 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 205 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:23.457381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:23.536304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:23.575263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b86d63-8a1c-4c31-acb6-ea7f025d8090 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:23.610645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:23.611346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.611626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.611906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.612737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=d1387a119eb04d0a9efe0164de9a8c3c) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:23.613043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:23.614483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50e01912-704b-4bd3-abb9-a40c5bfa23e1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:23.621026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 437/870] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:23 2022] GET /identity/v3/role_assignments?user.id=d1387a119eb04d0a9efe0164de9a8c3c&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:23.977687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be2d4a5-e7e0-4de1-a956-c281b85d2aff None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.980203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be2d4a5-e7e0-4de1-a956-c281b85d2aff None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.980203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6be2d4a5-e7e0-4de1-a956-c281b85d2aff None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:23.980203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 434/871] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:23.985717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e850c-5959-404f-ae69-47b55e787070 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:23.986112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e850c-5959-404f-ae69-47b55e787070 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:23.986487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a0e850c-5959-404f-ae69-47b55e787070 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:24.004149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0a0e850c-5959-404f-ae69-47b55e787070 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:24.005469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0a0e850c-5959-404f-ae69-47b55e787070 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:24.085571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 438/872] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:23 2022] POST /identity/v3/auth/tokens => generated 1151 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:24.100482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be2d4a5-e7e0-4de1-a956-c281b85d2aff None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:24.177838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:24.212027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6be2d4a5-e7e0-4de1-a956-c281b85d2aff None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:24.246552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58277865-0778-4566-a2dc-da284145babd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:24.247183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58277865-0778-4566-a2dc-da284145babd None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:24.247604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58277865-0778-4566-a2dc-da284145babd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:24.247883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58277865-0778-4566-a2dc-da284145babd None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:24.248614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58277865-0778-4566-a2dc-da284145babd None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:24.249974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58277865-0778-4566-a2dc-da284145babd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:24.265637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 435/873] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:24 2022] POST /identity/v3/services => generated 251 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:25.432560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d74ea8-86bb-4a70-959b-6f9eb43057e7 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.433483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d74ea8-86bb-4a70-959b-6f9eb43057e7 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.434003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d74ea8-86bb-4a70-959b-6f9eb43057e7 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.436267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 439/874] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:25 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:25.442004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.442433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.442614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.460407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:25.461693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:25.516566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf12cb-aad0-47ad-a821-5d8f314fbc1d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.516900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf12cb-aad0-47ad-a821-5d8f314fbc1d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.517126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cf12cb-aad0-47ad-a821-5d8f314fbc1d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.518725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 440/875] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:25.523721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.524014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.524299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.538909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 436/876] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:25 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:25.541763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:25.550796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:25.563591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:25.603159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:25.615445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 441/877] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:25 2022] POST /identity/v3/auth/tokens => generated 1252 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:25.627189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:25.639853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:25.669810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8221b6b-a180-4ebc-9cd5-3949a8489a57 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:25.706074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:25.706653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.706964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.707243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.718490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:25.727232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:25.728249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:25.729148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:25.733296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 437/878] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:25 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 166 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:25.734612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:25.750675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c9eba23-7ff1-449b-ac74-f04b7376c51f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:25.788519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:25.789188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.789474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.789913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.790636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:25.792096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddb3894-f390-41cd-9c6f-e15c6fe14e3d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:25.815301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 442/879] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:25 2022] POST /identity/v3/services => generated 269 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:25.817801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30951dc5-a5e2-4366-9884-eb77b98ac676 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.818613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30951dc5-a5e2-4366-9884-eb77b98ac676 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.819811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30951dc5-a5e2-4366-9884-eb77b98ac676 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.822611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 443/880] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:25 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:25.828961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.829556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.830038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.848837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f02dd523-a97c-4afe-aad6-8950472bdd11 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:25.854799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:25.856265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:25.881666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:25.908162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:25.909244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:25.909642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:25.910107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:25.911153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:25.911658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:25.913729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:25.924003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:25.927135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 438/881] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:25 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 193 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:25.934330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:25.961950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 444/882] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:25 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:25.979085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.067048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3ae119d-6462-4395-83c1-ed1be9472846 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.103568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.104446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.104732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.105079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.107582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12c2a2c2-0a16-4ee7-9a30-733b7917c7b2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.117201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.119085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:26.139667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:26.140917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:26.141968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:26.141968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:26.141968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:26.141968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:26.145607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 439/883] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:25 2022] GET /identity/v3/domains/Default => generated 87 bytes in 207 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:26.147265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.153588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.154610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.155081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.155502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.156327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.158366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:26.175342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:26.176553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:26.177537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 445/884] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:25 2022] GET /identity/v3/services/object-store => generated 93 bytes in 199 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:26.185044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.187082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.253107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.271925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0431eb-cd0e-4d38-8b8a-3f7fcb6bd8fd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.309297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.310110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.310490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.311139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.312062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:26.312492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.313964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.327427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb2c8fac-cc32-46b9-92ad-a24b5fefd9f5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.332173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 440/885] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:26.333592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.367669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.368490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.368887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.369261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.370190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:26.370459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.372156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c2b6fef-1197-465f-8152-ef17b2853d1e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.377247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 446/886] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:26.378589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57a914-aef1-414a-b99e-dd12e3c9d68a None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.379567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57a914-aef1-414a-b99e-dd12e3c9d68a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.379567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f57a914-aef1-414a-b99e-dd12e3c9d68a None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.380910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 447/887] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:26.383144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f57a914-aef1-414a-b99e-dd12e3c9d68a None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.437170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.453348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54ac9655-0f91-429d-b738-33096e6c4752 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.491706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.492917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.493464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.493899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.500709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f57a914-aef1-414a-b99e-dd12e3c9d68a None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.502713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.504399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.511514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-53b20890-05c3-4ddd-ad0f-52b116f10178 None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:26.523182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:26.524300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:26.525253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:26.525817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 441/888] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 192 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:26.526546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.526943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.527159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.542476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.543172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.543453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.543730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.544307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:26.544416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:26.544725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.545492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:26.546081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.550747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 448/889] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/services?type=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:26.551556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.653263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 442/890] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:26 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:26.676320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.690298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10465109-0eb7-4c0b-b227-2ce93a8abba0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.729045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.729879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.730253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.730627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.731591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:26.732009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.733542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.754750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 449/891] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:26.760904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.768731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.800201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb50bf77-e26b-4ee4-a8bc-f2127c0966cf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.801995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:26.844782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.845948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.846538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.847008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.848614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:26.850163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:26.866070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:26.867555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:26.868634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:26.876444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 443/892] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 196 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:26.878316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:26.888248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cde3aee-a1bd-4a23-8211-960185a0e5f9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:26.932048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:26.933131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:26.934138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:26.934656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:26.996876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a3cc82-b3d5-4b1a-ac76-9ed0c54bbd8e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.034094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.034799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.035108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.035382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.035575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.036085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:27.036392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.037703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.037829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.054505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 444/893] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:27.056084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.060886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.085011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.168783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 450/894] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:26 2022] GET /identity/v3/auth/tokens => generated 1362 bytes in 404 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:27.170743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.172564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-908034f2-bbc4-434c-8421-dbf5a816ad36 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.208299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.209341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.209911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.210409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.211371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:27.211890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.213471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.223972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 445/895] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:27.225102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.277962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e2ffe2f-ad1a-40ed-aaaa-9e0344d25f8f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.295484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.311638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.318546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.319888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.320506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.320876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.331608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.333065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.337155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7563fd8c-324f-47f5-b37f-e25384a9b1d5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:27.352933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:27.353876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:27.354761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:27.354761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:27.354761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:27.354761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:27.358848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 451/896] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:27.360096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.376307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.379044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.379577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.380074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.381098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] RBAC: Adding query filter params (interface=public, service_id=aab15475436b4e4ebca5725148000455, region_id=RegionOne) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:27.381617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.384428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.397145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 446/897] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/endpoints?service_id=aab15475436b4e4ebca5725148000455&interface=public®ion_id=RegionOne => generated 194 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:27.402189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.496149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4351d01c-3d91-4fcc-972a-29d4a5ba7183 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.517223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad7546d7-9b5a-4bb7-b018-8e8f4694a67a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.544861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.546363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.547152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.548031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.549248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:27.550013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.550909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.553723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.557889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.558420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.558707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.559030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.559711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.561438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.574292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 447/898] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/services => generated 1268 bytes in 167 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:48:27.581146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 452/899] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 217 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:27.582674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.695426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-666b7bde-b6df-4394-b234-cd8e2e214c35 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.739524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.740724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.741252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.741740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.750204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.752049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8b8b68c3-456d-4f95-a8f6-fead637c5cb4 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:27.768246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:27.769201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:27.770079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:27.773048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 448/900] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/projects/service => generated 88 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:27.779630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.833025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:27.896701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13b7fa5b-edd6-4828-b4e5-f476afc32e1d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:27.932682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:27.933341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.933624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.933897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.934677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:27.935012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:27.936694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:27.940254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad824a9-89c1-4ec8-abf1-794406a0bc68 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.940544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad824a9-89c1-4ec8-abf1-794406a0bc68 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.940822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad824a9-89c1-4ec8-abf1-794406a0bc68 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.942431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 449/901] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:27 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:27.948350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:27.948638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:27.948934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:27.954863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 453/902] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:27.957313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:27.967462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:27.968638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:28.043207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.062931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 450/903] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:27 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:28.075509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c0873a-48ec-4e06-95b9-74f950797711 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.077910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.085978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.128693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.129408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.129687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.130398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.131263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] RBAC: Adding query filter params (role.id=eb8e58e45a014ce2a398a53f365f1d99, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=f2b854f126714f82bfa98223c38e7376) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:28.132538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.134440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.145412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 454/904] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:27 2022] GET /identity/v3/role_assignments?user.id=f2b854f126714f82bfa98223c38e7376&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=eb8e58e45a014ce2a398a53f365f1d99 => generated 259 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:28.203997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ffc811-1072-4f39-b84b-5f3bca606731 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.244061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.244910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.245190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.245464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.246187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.247634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:28.261144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:28.262162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:28.263614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 451/905] 10.222.0.18 () {56 vars in 1132 bytes} [Sun Jul 24 00:48:28 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:28.271919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.382711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6cddb4-821b-4b09-8db7-388238fadc6e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.424669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.425736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.426238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.426729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.427679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:28.428212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.430740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.436658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 455/906] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:28 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:28.442578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.495905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.549603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9739b02-0412-4e9f-ae4b-6214615f2e14 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.589034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.589703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.589985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.590263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.590989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:28.591296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.592655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d386fe9f-884c-4a69-88c3-fea5f3886f84 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.597919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 452/907] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:28 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:28.603974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.637071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.699785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360a70f-a1f7-4045-b91b-8cad9c5355c1 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.700078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360a70f-a1f7-4045-b91b-8cad9c5355c1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.700364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7360a70f-a1f7-4045-b91b-8cad9c5355c1 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.701887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 453/908] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:28.708144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.708830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.710400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.716515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e6e2d47-6511-4286-80dd-1cae331d23b3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.732222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:28.733720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:28.753085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.753967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.754079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.754301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.755117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] RBAC: Adding query filter params (interface=public, service_id=bf23e0987c434b00acb81ba2de4df23a, region_id=RegionOne) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:28.755428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.757200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.762835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 456/909] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:28 2022] GET /identity/v3/endpoints?service_id=bf23e0987c434b00acb81ba2de4df23a&interface=public®ion_id=RegionOne => generated 194 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:28.767850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.813752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 454/910] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:28 2022] POST /identity/v3/auth/tokens => generated 1362 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:28.840720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:28.891145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.897499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2ea519-2530-44a4-b25e-c9b1a0f70475 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:28.906497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:28.938565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:28.939299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:28.939583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:28.939861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:28.940660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:28.942035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1b29a9-f5a7-43db-ae2b-81897fada2fa None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:28.950880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 457/911] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:28 2022] GET /identity/v3/services => generated 1268 bytes in 182 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:48:28.979416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a645380-88e8-41c6-b7cb-1909fdb51138 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:29.018904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:29.019850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:29.020288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:29.020717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:29.021658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:29.023519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bfbac29-dd1e-487f-9328-0f5e96305112 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:29.042986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 455/912] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:28 2022] POST /identity/v3/services => generated 241 bytes in 203 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:29.906289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cffd054-37fa-44b2-b0a1-8102caf8424c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:29.906832 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cffd054-37fa-44b2-b0a1-8102caf8424c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:29.906915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5cffd054-37fa-44b2-b0a1-8102caf8424c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:29.908519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 458/913] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:29.914713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:29.915152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:29.915341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:29.932121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:29.933312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:30.012671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 456/914] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:29 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:30.035555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cffd054-37fa-44b2-b0a1-8102caf8424c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.142304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5cffd054-37fa-44b2-b0a1-8102caf8424c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.177588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.178312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.178762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.179260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] PATH_INFO: `/v3/services/volumev3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.180152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] RBAC: Authorizing `identity:get_service(service_id=volumev3)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:30.181530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-bc0cbaae-9949-4d67-a5c7-2347efd9428a None admin] Could not find service: volumev3.: keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:30.193909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:30.195510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: volumev3. Jul 24 00:48:30.196372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:30.201260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 459/915] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/services/volumev3 => generated 89 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:30.203879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.235843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:30.312508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-33d02825-6a91-4e10-a1e8-d5ab25355ed3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.315717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad9df-3a4c-49dc-a126-6679fad17f15 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.315854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad9df-3a4c-49dc-a126-6679fad17f15 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.316256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d18ad9df-3a4c-49dc-a126-6679fad17f15 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.317979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 460/916] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:30.324823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.325412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.325551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.344030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:30.345298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:30.350062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.351127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.351625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.352081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.352966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] RBAC: Adding query filter params (name=volumev3) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:30.353455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:30.355119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:30.360454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 457/917] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/services?name=volumev3 => generated 125 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:30.367857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.370014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:30.435486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 461/918] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:30 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:30.455040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.505020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:30.515359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67b88484-aca0-4810-92a3-f52b7dab8cd4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.562053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.563688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.564164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.564627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.565539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] RBAC: Adding query filter params (type=volumev3) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:30.565647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cae5fcd-31d6-4c2c-8d2c-f15941d88ce8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.566132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:30.568823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:30.576644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 458/919] 10.222.0.18 () {56 vars in 1114 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/services?type=volumev3 => generated 362 bytes in 209 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:30.582289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.607430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.608581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.609086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.609618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.627223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:30.642002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:30.643276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:30.644128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:30.646849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 462/920] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 189 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:30.650485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.674690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:30.716094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af9fea20-3677-46b1-9373-0ddbaf4a4162 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.762487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.763185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.763475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.763761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.764946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:30.766644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:30.791798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 459/921] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:30 2022] POST /identity/v3/endpoints => generated 346 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:30.805057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df47b29-3151-4191-92d0-d68b24d131db None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:30.852363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:30.853101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:30.853379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:30.853648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:30.854452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:30.854759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:30.856149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:30.866698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 463/922] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 216 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:30.873264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:30.891889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:30.995087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c7fe1f1-1a4a-4092-b4da-ec0a03a885e6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.032680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.033874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.034378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.034880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.048140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.049928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-284b3bfe-4642-44d3-aaa8-336a76879295 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:31.068765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:31.070763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:31.071676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:31.071676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:31.071676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:31.071676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:31.078512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 460/923] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:30 2022] GET /identity/v3/domains/Default => generated 87 bytes in 201 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:31.080074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.161682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.188848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec1e44-6e39-44ba-95d9-32bd73835b3f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.189161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec1e44-6e39-44ba-95d9-32bd73835b3f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.189454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ec1e44-6e39-44ba-95d9-32bd73835b3f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.190976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 461/924] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:31.191615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a346904a-af68-406b-a40a-e49df607a16f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.197230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.197516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.197793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.216939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:31.218249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:31.228399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.229063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.229345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.229619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.230337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:31.230641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.232065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.249994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 464/925] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:31.251416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.286987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.296195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 462/926] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:31 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:31.321198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.363993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63ab5557-fa80-45ab-82fd-234a317816a4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.395574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.402209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.403158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.403449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.403722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.411048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.412506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-6d80d904-7c02-4c09-97a3-128fbf4d1efc None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:31.428777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:31.429742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:31.430668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 465/927] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:31.431873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65057e5-4c5c-40ae-902b-94caeaf6faae None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.431987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65057e5-4c5c-40ae-902b-94caeaf6faae None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.432229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e65057e5-4c5c-40ae-902b-94caeaf6faae None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.433624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 466/928] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:31.437288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65057e5-4c5c-40ae-902b-94caeaf6faae None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.441939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60313859-582a-4547-aa65-8c01aadc4aab None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.479663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.480480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.480769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.481048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] PATH_INFO: `/v3/services/compute_legacy` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.481697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] RBAC: Authorizing `identity:get_service(service_id=compute_legacy)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.483095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-3bb51003-50ed-46d3-a3b0-34f727f3312b None admin] Could not find service: compute_legacy.: keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:31.496289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:31.497201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute_legacy. Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:31.498055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 463/929] 10.222.0.18 () {56 vars in 1132 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/services/compute_legacy => generated 95 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:31.499807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.499807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.499807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.518341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:31.519484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:31.556086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e65057e5-4c5c-40ae-902b-94caeaf6faae None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.577225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.590816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 464/930] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:31 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:31.590816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.602996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.603790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.604071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.604350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.605202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:31.605505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.607260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.632417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 467/931] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:31.632417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.686402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.709364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b1b6b6-57e1-44ad-b21c-4ac980fb059d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.738719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca55271-0352-40c8-9e64-5b6014861733 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.749942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.750917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.751423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.751891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.752797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] RBAC: Adding query filter params (name=compute_legacy) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:31.753294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.754861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.760844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 465/932] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/services?name=compute_legacy => generated 131 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:31.761494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.779680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.780705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.780985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.781434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.782359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.783756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:31.796543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:31.797653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:31.798637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 468/933] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/services/object-store => generated 93 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:31.799514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.825664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.873165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fc20e7-ddf7-4d4e-a691-9aa5313a6e4d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.909973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.910786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.911075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.911353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.911985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f054bf1d-55c3-4978-9787-88d0ea34b41d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:31.923492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.924921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.927733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:31.944105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:31.945725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:31.946709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:31.946709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:31.946709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:31.946709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:31.946709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 466/934] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/domains/Default => generated 87 bytes in 185 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:31.947418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:31.957759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:31.958893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:31.959245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:31.959624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:31.960667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] RBAC: Adding query filter params (type=compute_legacy) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:31.961134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:31.962816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:31.968411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 469/935] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/services?type=compute_legacy => generated 386 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:31.969304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.067843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e38e1bf-f00e-4b82-9c07-3941c9f4a467 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.076083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.082289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c7cfa7-474a-4fdf-9d0a-c3d4252a5e81 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.105754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.106542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.106961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.107240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.108092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:32.108401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.109974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.120955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 467/936] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:32.120955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.122566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.123268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.123555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.123834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.124563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:32.125320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.126712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.140759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 470/937] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:31 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 172 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:32.146012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.188734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.224336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-431173d5-f535-4a8c-b9e9-689cb39c8d79 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.255082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b2942d6-cd57-47c6-a17a-7fb2a1b489c7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.261150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.261815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.262103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.262383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.263151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.264523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.284023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 468/938] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:32 2022] POST /identity/v3/endpoints => generated 347 bytes in 168 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:32.284947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.315166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.315964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.316556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.316805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.317584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:32.317983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.320246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.326822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 471/939] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:32 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:32.336903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.339088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.451878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7ac6524-e9c9-446a-becb-a5fc79b79cff None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.483236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.487523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cf0e247-76e1-4e6f-ae56-de330b9dc809 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.500895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.501713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.502002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.502311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.510551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.512048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:32.530161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:32.531176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:32.532074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:32.532525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 469/940] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:32 2022] GET /identity/v3/projects/service => generated 88 bytes in 247 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:32.536706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.536706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.536706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.536706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.537060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] RBAC: Adding query filter params (interface=public, service_id=ecbc60e269fa45d88f3060fdb1bf872e, region_id=RegionOne) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:32.537060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.538610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.539619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.544432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 472/941] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:32 2022] GET /identity/v3/endpoints?service_id=ecbc60e269fa45d88f3060fdb1bf872e&interface=public®ion_id=RegionOne => generated 194 bytes in 208 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:32.549835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.651350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.659141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc51b3af-ea94-4510-bef2-fdbc86adb5e2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.669283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d36e1ca-f69f-4f72-bbdd-559c3475b60b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.697101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.697939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.698437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.698939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.699743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.701327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.711214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.711895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.712181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.712558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.713397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:32.713715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.715075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 473/942] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:32 2022] GET /identity/v3/services => generated 1497 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 00:48:32.715223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d811cb2-d2bb-4099-a78a-2f8c4b84ae58 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.729944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 470/943] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:32 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 191 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:32.736114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:32.768244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:32.859008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d68610f-7fe7-4672-9364-83762f3a99ab None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:32.902958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:32.903778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.904061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.904341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/f2b854f126714f82bfa98223c38e7376/roles/eb8e58e45a014ce2a398a53f365f1d99` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.906564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=f2b854f126714f82bfa98223c38e7376, role_id=eb8e58e45a014ce2a398a53f365f1d99)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:32.908048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5343141b-4468-4016-b20d-b05a4c3e4ca5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:32.923370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 474/944] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:32 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/f2b854f126714f82bfa98223c38e7376/roles/eb8e58e45a014ce2a398a53f365f1d99 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:32.963942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb65f8d6-e2d2-449a-a049-dc952efa75c2 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.964235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb65f8d6-e2d2-449a-a049-dc952efa75c2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.964504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb65f8d6-e2d2-449a-a049-dc952efa75c2 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.966175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 471/945] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:32.972189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:32.972491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:32.972748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:32.990984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:32.992295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:33.068326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 475/946] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:32 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:33.093770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb65f8d6-e2d2-449a-a049-dc952efa75c2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:33.200301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb65f8d6-e2d2-449a-a049-dc952efa75c2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:33.239475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:33.240148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:33.240434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:33.240712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:33.242004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:33.243481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:33.244863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING keystone.server.flask.common [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] Deprecated: Not specifying a domain during a create user, group or project call, and relying on falling back to the default domain, is deprecated as of Liberty. There is no plan to remove this compatibility, however, future API versions may remove this, so please specify the domain explicitly or use a domain-scoped token. Jul 24 00:48:33.264355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.sql.core [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] Conflict project: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 24 00:48:33.264355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 24 00:48:33.264355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [parameters: {'id': 'f2eb350e2f7c42308b96fd644b06148f', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 24 00:48:33.264355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80232) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f2eb350e2f7c42308b96fd644b06148f.: keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f2eb350e2f7c42308b96fd644b06148f. Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:48:33.272480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 241, in create_project Jul 24 00:48:33.273290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return project_ref.to_dict() Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application yield resource Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.rollback() Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.force_reraise() Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise self.value Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application session.commit() Jul 24 00:48:33.274231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._prepare_impl() Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.flush() Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._flush(objects) Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application compat.raise_( Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application flush_context.execute() Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 24 00:48:33.275159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rec.execute(self) Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1097, in _emit_insert_statements Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application c = connection._execute_20( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 24 00:48:33.276043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:48:33.276831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-service' for key 'project.ixu_project_name_domain_id'") Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [SQL: INSERT INTO project (id, name, domain_id, description, enabled, extra, parent_id, is_domain) VALUES (%(id)s, %(name)s, %(domain_id)s, %(description)s, %(enabled)s, %(extra)s, %(parent_id)s, %(is_domain)s)] Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [parameters: {'id': 'f2eb350e2f7c42308b96fd644b06148f', 'name': 'service', 'domain_id': 'default', 'description': '', 'enabled': 1, 'extra': '{}', 'parent_id': 'default', 'is_domain': 0}] Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:33.277681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 216, in create_project Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ret = self.driver.create_project(project_id, project) Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - Duplicate entry found with name service at domain ID default. Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:33.278458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 176, in post Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.resource_api.create_project( Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 218, in create_project Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.Conflict( Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store project - it is not permitted to have two projects with either the same name or same id in the same domain: name is service, project id f2eb350e2f7c42308b96fd644b06148f. Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:33.281649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 472/947] 10.222.0.18 () {60 vars in 1140 bytes} [Sun Jul 24 00:48:33 2022] POST /identity/v3/projects => generated 262 bytes in 181 msecs (HTTP/1.1 409) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:33.282530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:33.307214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:33.386104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7bb4cfc-23e4-4fcb-bdc3-ba1e10c87239 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:33.422739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:33.423626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:33.423909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:33.424178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:33.431706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:33.433195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ed768023-3043-482e-91b5-60c68c98f869 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:33.447743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:33.448657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:33.449965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 476/948] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:33 2022] GET /identity/v3/projects/service => generated 88 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:33.455229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:33.458404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:33.564672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7d6fcaa-48fa-42ec-aaaf-968b056fe4c0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:33.600824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:33.601551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:33.601838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:33.602116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:33.602900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:33.603210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:33.604838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-086c6a02-e189-4ddc-aa09-e214a0ebc6fe None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:33.625491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 473/949] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:33 2022] GET /identity/v3/projects?name=service => generated 411 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:34.591356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baec1ea5-a6f5-4e90-a930-c1192f2ad862 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:34.591879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baec1ea5-a6f5-4e90-a930-c1192f2ad862 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:34.592189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-baec1ea5-a6f5-4e90-a930-c1192f2ad862 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:34.593907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 477/950] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:34.599283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d72184c-df5f-49aa-be77-649ee65ee3d5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:34.599708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d72184c-df5f-49aa-be77-649ee65ee3d5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:34.600034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d72184c-df5f-49aa-be77-649ee65ee3d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:34.618618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2d72184c-df5f-49aa-be77-649ee65ee3d5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:34.620156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2d72184c-df5f-49aa-be77-649ee65ee3d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:34.680215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:34.691785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 474/951] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:34.713750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baec1ea5-a6f5-4e90-a930-c1192f2ad862 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:34.797636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:34.825547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-baec1ea5-a6f5-4e90-a930-c1192f2ad862 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:34.859728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a640ab-4fc1-47b1-bf25-3e7be63effe5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:34.860034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a640ab-4fc1-47b1-bf25-3e7be63effe5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:34.860300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5a640ab-4fc1-47b1-bf25-3e7be63effe5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:34.861257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:34.862058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:34.862339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:34.862618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:34.863307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:34.864680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:34.865141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 475/952] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:34 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:34.867888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:34.868183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:34.868466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:34.879562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:34.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:34.882133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 478/953] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:34 2022] GET /identity/v3/services/compute => generated 88 bytes in 168 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:34.887693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:34.893113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:34.894426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:34.978213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 476/954] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:34 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:35.001643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.016126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c382e885-8ff5-4adf-8356-f59f15ea4d85 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.056381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.057285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.057399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.057595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.058299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:35.058930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.060303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36703b04-3340-46b5-b2f9-7175aed609a2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.066942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 479/955] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:34 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:35.066942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115179b-5b65-4287-a4a8-aa2b7b0ff689 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.066942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115179b-5b65-4287-a4a8-aa2b7b0ff689 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.067242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5115179b-5b65-4287-a4a8-aa2b7b0ff689 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.068648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 480/956] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:35.070971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115179b-5b65-4287-a4a8-aa2b7b0ff689 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.078223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.118343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24c00e9c-79b7-4708-b3a6-b0ea0d5e1593 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.140572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.153767 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.154653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.154992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.155323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.155979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.157783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-3d3b922e-adfd-43dd-ab67-433906d50de8 None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:35.171763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:35.172921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:35.173820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 477/957] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/services/object-store => generated 93 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:35.174602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.174602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.175422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.192141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:35.192300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5115179b-5b65-4287-a4a8-aa2b7b0ff689 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.193327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:35.230366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.230985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.231299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.231544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.232635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] RBAC: Adding query filter params (type=compute) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:35.232635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.233952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.246802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 481/958] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/services?type=compute => generated 124 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:35.246802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.278515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 478/959] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:35.310104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.352001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.376351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff31196-bc2a-4e0d-91fd-0f0923841c4a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.425989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.426662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.426977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.427252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.427958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:35.428811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.430185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.437840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 482/960] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 198 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:35.443200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.445918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4118c89-284c-4356-8a63-20ade51cf25f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.479563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.497429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.498533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.499010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.499486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.510926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:35.519543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:35.520520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:35.521401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:35.524668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 479/961] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 215 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:35.530901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.566420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-597869be-1a91-43d8-a012-b53d2efe46a4 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.597920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.603686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.604772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.605439 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.605719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.606428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:35.606733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.608184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.618822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 483/962] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:35.619895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.664723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a54db3f9-a83c-4b0e-8615-5097065a36c9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.702130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.702841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.703402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.703402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.704137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:35.704446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.705829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.718818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 480/963] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:35.719910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.755764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7f28aa2-25fa-48ce-a9db-977d94e07994 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.769620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.795166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.795831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.796111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.796384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.797429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.799615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f5d7605-7d18-4bfb-93c9-e60250727afd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.827081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 484/964] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:35 2022] POST /identity/v3/endpoints => generated 349 bytes in 207 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:35.832339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8de1f3-6a83-4ef2-99d4-a0c7c3b18657 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.832671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8de1f3-6a83-4ef2-99d4-a0c7c3b18657 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.832944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a8de1f3-6a83-4ef2-99d4-a0c7c3b18657 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.834644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 485/965] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:35.851174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.851174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.851174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.856160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b606e3f2-55a2-47c6-987a-a944ccadd56c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:35.864600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:35.865358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:35.903994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:35.904854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:35.905139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:35.905411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:35.921130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:35.922601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:35.927759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:35.955132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:35.956126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:35.957492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:35.957492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:35.957492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:35.957492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:35.963230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 486/967] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:35 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:35.963539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 481/967] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/domains/Default => generated 87 bytes in 240 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:35.966360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:35.983291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.097263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:36.105230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d75b450-2ec2-4a07-8e11-a8341afeefed None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.116659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2462c989-9947-4a48-bbd7-c132f7472bdd None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.151145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.152090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.152373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.152898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.154027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:36.154605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.156170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.158551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.159792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.160079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.160352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.161065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.162470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.168833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 487/968] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 203 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:36.174542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.185810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.sql.core [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] Conflict user: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 24 00:48:36.185810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 24 00:48:36.185810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [parameters: {'user_id': '204c3182a560403cb3f651f59a412e92', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 24 00:48:36.185810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: (Background on this error at: https://sqlalche.me/e/14/gkpj) {{(pid=80232) wrapper /opt/stack/keystone/keystone/common/sql/core.py:533}} Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default.: keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:48:36.194312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application pymysql.err.IntegrityError: (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application The above exception was the direct cause of the following exception: Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 528, in wrapper Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return method(*args, **kwargs) Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 141, in create_user Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return base.filter_user(user_ref.to_dict()) Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:48:36.195635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1060, in _transaction_scope Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application yield resource Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application next(self.gen) Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 662, in _session Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.rollback() Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 227, in __exit__ Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.force_reraise() Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_utils/excutils.py", line 200, in force_reraise Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise self.value Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 659, in _session Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._end_session_transaction(self.session) Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 687, in _end_session_transaction Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application session.commit() Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 1451, in commit Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._transaction.commit(_to_root=self.future) Jul 24 00:48:36.196564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 829, in commit Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._prepare_impl() Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 808, in _prepare_impl Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.session.flush() Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3383, in flush Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._flush(objects) Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3523, in _flush Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application transaction.rollback(_capture_exception=True) Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/langhelpers.py", line 70, in __exit__ Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application compat.raise_( Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/session.py", line 3483, in _flush Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application flush_context.execute() Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 456, in execute Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rec.execute(self) Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/unitofwork.py", line 630, in execute Jul 24 00:48:36.197453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.preloaded.orm_persistence.save_obj( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application _emit_insert_statements( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/orm/persistence.py", line 1238, in _emit_insert_statements Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = connection._execute_20( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1631, in _execute_20 Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return meth(self, args_10style, kwargs_10style, execution_options) Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/sql/elements.py", line 332, in _execute_on_connection Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return connection._execute_clauseelement( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1498, in _execute_clauseelement Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ret = self._execute_context( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1862, in _execute_context Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._handle_dbapi_exception( Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 2041, in _handle_dbapi_exception Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application util.raise_(newraise, with_traceback=exc_info[2], from_=e) Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/util/compat.py", line 208, in raise_ Jul 24 00:48:36.198431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1819, in _execute_context Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self.dialect.do_execute( Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 732, in do_execute Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application cursor.execute(statement, parameters) Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 148, in execute Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result = self._query(query) Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/cursors.py", line 310, in _query Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application conn.query(q) Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 548, in query Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._affected_rows = self._read_query_result(unbuffered=unbuffered) Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 775, in _read_query_result Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application result.read() Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 1156, in read Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application first_packet = self.connection._read_packet() Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/connections.py", line 725, in _read_packet Jul 24 00:48:36.199312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application packet.raise_for_error() Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/protocol.py", line 221, in raise_for_error Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application err.raise_mysql_exception(self._data) Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/pymysql/err.py", line 143, in raise_mysql_exception Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise errorclass(errno, errval) Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application oslo_db.exception.DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, "Duplicate entry 'default-cinder' for key 'local_user.domain_id'") Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [SQL: INSERT INTO local_user (user_id, domain_id, name, failed_auth_count, failed_auth_at) VALUES (%(user_id)s, %(domain_id)s, %(name)s, %(failed_auth_count)s, %(failed_auth_at)s)] Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [parameters: {'user_id': '204c3182a560403cb3f651f59a412e92', 'domain_id': 'default', 'name': 'cinder', 'failed_auth_count': 0, 'failed_auth_at': None}] Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application (Background on this error at: https://sqlalche.me/e/14/gkpj) Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:36.200140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 244, in post Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.create_user( Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1000, in create_user Jul 24 00:48:36.200922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = self._create_user_with_federated_objects(user, driver) Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 966, in _create_user_with_federated_objects Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application user = driver.create_user(user['id'], user) Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/sql/core.py", line 563, in wrapper Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.Conflict(type=conflict_type, Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.Conflict: Conflict occurred attempting to store user - Duplicate entry found with name cinder at domain ID default. Jul 24 00:48:36.201869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.205211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 482/969] 10.222.0.18 () {60 vars in 1131 bytes} [Sun Jul 24 00:48:35 2022] POST /identity/v3/users => generated 160 bytes in 222 msecs (HTTP/1.1 409) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:36.214872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.230101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:36.310914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5e8a297-f70f-448c-af4b-572b804abf2f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.339851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5738cb9d-ba8a-405a-a792-bc80e6dd24c0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.348033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.348785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.349066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.349341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] PATH_INFO: `/v3/users/glance-swift` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.357470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] RBAC: Authorizing `identity:get_user(user_id=glance-swift)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.358952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] Could not find user: glance-swift.: keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:36.374826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:36.376912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: glance-swift. Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:36.378181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 488/970] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/users/glance-swift => generated 90 bytes in 202 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:36.378985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.378985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.378985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.379240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] PATH_INFO: `/v3/users/cinder` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.385842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.392226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] RBAC: Authorizing `identity:get_user(user_id=cinder)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.394261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] Could not find user: cinder.: keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:36.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:36.410886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: cinder. Jul 24 00:48:36.411824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:36.421336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 483/971] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/users/cinder => generated 84 bytes in 205 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:36.422577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.435257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:36.519162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e372f68-04bb-47ee-8fe5-c5fe7f60907b None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.521271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:36.543337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f67078c2-9c6f-4e9b-83cb-b2bac4b64ce9 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.569500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.570528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.571234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.571742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.574016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] RBAC: Adding query filter params (domain_id=default, name=glance-swift) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:36.574832 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.576994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.585580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.586526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.587041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.587523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.588542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] RBAC: Adding query filter params (name=cinder) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:36.589070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.590709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.597564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 489/972] 10.222.0.18 () {56 vars in 1149 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/users?domain_id=default&name=glance-swift => generated 384 bytes in 212 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:36.603234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:36.632252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 484/973] 10.222.0.18 () {56 vars in 1101 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/users?name=cinder => generated 354 bytes in 210 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:36.722935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef257e66-eed8-4410-8677-70cb02f23ecf None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:36.762050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:36.762716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:36.763066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:36.763308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:36.795411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:36.839298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:36.840765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:36.946702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 490/974] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/auth/tokens => generated 1464 bytes in 344 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:48:36.952670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.019229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:37.057461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5774769b-86b5-4940-908a-aa54168ede39 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.091797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.092598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.092883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.093158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.103865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.105341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:37.123253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:37.124380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:37.129924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:37.129924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:37.129924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:37.129924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:37.129924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 485/975] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:37.132142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.191875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:37.233687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a8bd482-0f99-4485-a30e-ce2cdd2e8ab7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.267169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.267974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.268353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.268728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.269574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:37.269980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.271520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85124100-c017-4bcc-9018-5860769f1bd7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.283467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 491/976] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:37.289317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.387894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57196b19-1397-40f2-b818-ca7d9c31004b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.403551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbac9ab-886e-4788-8280-fb4959ca44a6 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.403860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbac9ab-886e-4788-8280-fb4959ca44a6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.404116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cbac9ab-886e-4788-8280-fb4959ca44a6 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.405493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 492/977] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:37.409790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:37.410635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.411030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.411249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.431180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:37.432372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:37.434605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.435772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.436141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.436484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.443545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.445474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:37.460147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:37.461073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:37.461986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 486/978] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:37 2022] GET /identity/v3/projects/service => generated 88 bytes in 172 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:37.467088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.518881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 493/979] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:37 2022] POST /identity/v3/auth/tokens => generated 1464 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:37.533855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.572113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3d4bde3-c0ce-4c11-83cc-368d727c34c2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.591864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:37.610307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.610988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.611275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.611558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.612335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:37.612646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.614066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.627796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 487/980] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:37 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:37.634168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:37.652500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9fcccc9-f274-4226-af22-cdd5f8db4167 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.676902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:37.688626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.689304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.689591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.689870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.690595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.692005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ebcef6a-a98e-4430-868f-a221434ba75a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.705813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 494/981] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:37 2022] POST /identity/v3/services => generated 242 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:37.753930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7360630-a7c9-41c7-867b-fc9cbe7dddf2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:37.800480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:37.801152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.801348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.801624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.802455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] RBAC: Adding query filter params (role.id=eb8e58e45a014ce2a398a53f365f1d99, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=f2b854f126714f82bfa98223c38e7376) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:37.802800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:37.804586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ba9102-b313-4b4a-9289-0cf1d0a10f09 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:37.819313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 488/982] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:37 2022] GET /identity/v3/role_assignments?user.id=f2b854f126714f82bfa98223c38e7376&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=eb8e58e45a014ce2a398a53f365f1d99 => generated 608 bytes in 178 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:37.978287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d33304-f55f-4b0b-a520-3f9b95845315 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.978625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d33304-f55f-4b0b-a520-3f9b95845315 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.978913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09d33304-f55f-4b0b-a520-3f9b95845315 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:37.980437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 495/983] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:37 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:37.985841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:37.986295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:37.986441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.004125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:38.005656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:38.077193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 489/984] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:37 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:38.099902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d33304-f55f-4b0b-a520-3f9b95845315 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:38.206052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09d33304-f55f-4b0b-a520-3f9b95845315 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:38.217123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:38.242801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:38.243623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:38.243909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:38.244188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.244844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:38.246218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:38.258701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:38.259875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:38.260753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:38.262871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 496/985] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:38 2022] GET /identity/v3/services/object-store => generated 93 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:38.266052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:38.294001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:38.369137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23dea576-6e97-4c9f-89db-a2a2a8348524 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:38.407528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:38.408200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:38.408488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:38.408769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.409476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:38.409786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:38.414599 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:38.420899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 490/986] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:38 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:38.426644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:38.528152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08e3f650-1fab-410b-85e4-2dc229a5d73f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:38.562791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:38.563475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:38.563761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:38.564040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.564819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:38.565055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:38.566434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:38.571787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 497/987] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:38 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:38.577583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:38.668552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:38.678116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0fc1464-3a0f-4d85-ae1e-0f72af98da70 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:38.712217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:38.712881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:38.713169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:38.713448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.714240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] RBAC: Adding query filter params (interface=admin, service_id=ecbc60e269fa45d88f3060fdb1bf872e, region_id=RegionOne) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:38.714558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:38.715957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f34b29a7-cffb-4414-94cf-2ae2dde55e4d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:38.721208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 491/988] 10.222.0.18 () {56 vars in 1249 bytes} [Sun Jul 24 00:48:38 2022] GET /identity/v3/endpoints?service_id=ecbc60e269fa45d88f3060fdb1bf872e&interface=admin®ion_id=RegionOne => generated 193 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:38.727381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:38.766807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:38.829978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b3957e4-82b5-4f72-9f78-47b038b66277 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:38.866618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:38.867470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:38.867758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:38.868036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:38.868722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:38.870105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6b2fa41-fd09-45b3-862a-38bd724bfca8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:38.877256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 498/989] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:38 2022] GET /identity/v3/services => generated 1727 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:48:39.784941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be76afcd-5fa6-4688-bb42-c451c05a0733 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:39.785456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be76afcd-5fa6-4688-bb42-c451c05a0733 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:39.785456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be76afcd-5fa6-4688-bb42-c451c05a0733 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:39.786969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 492/990] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:39.791939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea26011-4ce8-4a83-ad83-4eb994175df1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:39.792337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea26011-4ce8-4a83-ad83-4eb994175df1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:39.792508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fea26011-4ce8-4a83-ad83-4eb994175df1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:39.808763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-fea26011-4ce8-4a83-ad83-4eb994175df1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:39.809939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fea26011-4ce8-4a83-ad83-4eb994175df1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:39.883089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 499/991] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:39.900526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be76afcd-5fa6-4688-bb42-c451c05a0733 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:39.902861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11f1d6-e93b-40e3-9d2e-e43f8d882068 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:39.903157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11f1d6-e93b-40e3-9d2e-e43f8d882068 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:39.903570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee11f1d6-e93b-40e3-9d2e-e43f8d882068 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:39.904928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 500/992] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:39.909847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:39.910134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:39.910410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:39.927248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:39.928716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:39.955875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:39.999572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 501/993] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:39 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:40.022542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.026723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be76afcd-5fa6-4688-bb42-c451c05a0733 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.061089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.061800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.062079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.062362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.063049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.064387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:40.077168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:40.078219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:40.079106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 493/994] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:39 2022] GET /identity/v3/services/compute => generated 88 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:40.084017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.108295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:40.136755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef17fb90-5445-4974-b9bc-a83d4a7cbb13 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.173550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.174445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.174725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.175028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] PATH_INFO: `/v3/services/image` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.175675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.177163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.189000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e73ae93-d232-4a36-b7fe-42e9a32a4c23 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:40.190089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:40.191082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:40.191932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 502/995] 10.222.0.18 () {56 vars in 1105 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services/image => generated 86 bytes in 170 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:40.197997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.227703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:40.230604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.231390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.231624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.231953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.232648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] RBAC: Adding query filter params (name=compute) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:40.232962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.234327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.245182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 494/996] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:40.246743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.306627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb9d9bb7-691a-405a-8435-aada02763948 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.333591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:40.344508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.345160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.345447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.345726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.346431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] RBAC: Adding query filter params (name=image) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:40.346784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.348333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.350749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d4164d-553b-471f-9de1-2bdab3b80688 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.353552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 503/997] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services?name=image => generated 122 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:40.359094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.385211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.385936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.386168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.386448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.387183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] RBAC: Adding query filter params (type=compute) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:40.387493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.388866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.397232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 495/998] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:40.403238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.469712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:40.481294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1756edf-e57f-4c34-afdd-015d14cce5c2 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.511138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea3fdab-b083-4850-bddb-55deb938319d None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.519144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.519817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.520113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.520740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.521454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] RBAC: Adding query filter params (type=image) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:40.522137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.523822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.529087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 504/999] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services?type=image => generated 122 bytes in 170 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:40.556221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.557209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.557647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.558087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.559086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] RBAC: Adding query filter params (interface=public, service_id=ff27f2fd95774cd3b1d0c96b57e99dc2, region_id=RegionOne) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:40.559557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.561400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7fd094e-bf77-4d6d-bc48-68c388241eed None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.568075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 496/1000] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/endpoints?service_id=ff27f2fd95774cd3b1d0c96b57e99dc2&interface=public®ion_id=RegionOne => generated 194 bytes in 165 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:40.573903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:40.684014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe5ab1b7-8a45-4f4a-a4b7-5fa824028f61 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:40.719944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:40.720761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.721137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.721526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.722296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:40.723783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff79550-5f9e-4689-b03a-1a25b4e7291b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:40.731152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 505/1001] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:40 2022] GET /identity/v3/services => generated 1727 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:48:40.937748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c839a0f-9df0-4ffb-99ed-bda9105a02a4 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.938245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c839a0f-9df0-4ffb-99ed-bda9105a02a4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.938245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c839a0f-9df0-4ffb-99ed-bda9105a02a4 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.939638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 497/1002] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:40.945069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:40.945478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:40.945624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:40.962919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:40.964104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:40.978876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:41.033469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 506/1003] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:40 2022] POST /identity/v3/auth/tokens => generated 1560 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:41.056620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c839a0f-9df0-4ffb-99ed-bda9105a02a4 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:41.072726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:41.161864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c839a0f-9df0-4ffb-99ed-bda9105a02a4 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:41.197238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:41.197967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:41.198256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:41.198534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] PATH_INFO: `/v3/services/object-store` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:41.199214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] RBAC: Authorizing `identity:get_service(service_id=object-store)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:41.200575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] Could not find service: object-store.: keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:41.213288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:41.214280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: object-store. Jul 24 00:48:41.215165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:41.219083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 498/1004] 10.222.0.18 () {56 vars in 1126 bytes} [Sun Jul 24 00:48:41 2022] GET /identity/v3/services/object-store => generated 93 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:41.220445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:41.319060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2ced8a2-a711-4349-87d9-c430e5385677 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:41.352037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:41.352692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:41.352970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:41.353242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:41.353939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] RBAC: Adding query filter params (name=object-store) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:41.354243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:41.355638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:41.360454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 507/1005] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:41 2022] GET /identity/v3/services?name=object-store => generated 129 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:41.366456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:41.451236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:41.474471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d84c41b-2b3f-4c64-8290-c13eadf2b5aa None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:41.508773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:41.509438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:41.509724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:41.510003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:41.510701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] RBAC: Adding query filter params (type=object-store) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:41.511039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:41.512407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fb4c245-f4a9-407d-bce8-0e1a4dd0dc1d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:41.517475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 499/1006] 10.222.0.18 () {56 vars in 1122 bytes} [Sun Jul 24 00:48:41 2022] GET /identity/v3/services?type=object-store => generated 356 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:41.522973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:41.538414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:41.628034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-002e9bcf-8832-4760-8251-924532893c93 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:41.665380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:41.666044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:41.666321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:41.666591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:41.667368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:41.668734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed615d28-a98e-4737-bce8-49a4ef0d43f0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:41.690807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 508/1007] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:41 2022] POST /identity/v3/endpoints => generated 325 bytes in 162 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:42.584858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918c7b2-6735-4fbb-8a59-6a7151ce72a4 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:42.585375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918c7b2-6735-4fbb-8a59-6a7151ce72a4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:42.585453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f918c7b2-6735-4fbb-8a59-6a7151ce72a4 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:42.586965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 500/1008] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:42.591949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac44c1-1cb6-4c3b-84a6-bff7c623b843 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:42.592330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac44c1-1cb6-4c3b-84a6-bff7c623b843 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:42.592515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebac44c1-1cb6-4c3b-84a6-bff7c623b843 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:42.609196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ebac44c1-1cb6-4c3b-84a6-bff7c623b843 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:42.610387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ebac44c1-1cb6-4c3b-84a6-bff7c623b843 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:42.677566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:42.685764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 509/1009] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:42 2022] POST /identity/v3/auth/tokens => generated 1707 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:42.708824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f918c7b2-6735-4fbb-8a59-6a7151ce72a4 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:42.813893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f918c7b2-6735-4fbb-8a59-6a7151ce72a4 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:42.824474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac56f0d5-5848-448d-84a8-eb5a347b8bbe None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:42.824775 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac56f0d5-5848-448d-84a8-eb5a347b8bbe None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:42.825035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac56f0d5-5848-448d-84a8-eb5a347b8bbe None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:42.826529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 510/1010] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:42.831146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:42.831433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:42.831717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:42.848545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:42.849753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:42.850602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:42.851469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:42.851948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:42.852374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:42.853253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:42.854764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:42.864835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:42.879048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 501/1011] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:42 2022] POST /identity/v3/services => generated 242 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:42.934266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 511/1012] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:42 2022] POST /identity/v3/auth/tokens => generated 1803 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:42.957344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:43.069659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ec5d3ea8-a531-4484-8e95-6e505782a339 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:43.102874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:43.103676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.103961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.104237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] PATH_INFO: `/v3/services/compute` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.104885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] RBAC: Authorizing `identity:get_service(service_id=compute)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:43.106249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] Could not find service: compute.: keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:43.118671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:43.119682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: compute. Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:43.120539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 502/1013] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:42 2022] GET /identity/v3/services/compute => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:43.127683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:43.231018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e9878c0-2952-4ae5-b01c-0acc53accb3b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:43.255784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:43.268056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:43.268730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.269012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.269285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.269986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] RBAC: Adding query filter params (name=compute) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:43.270288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:43.271918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:43.277251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 512/1014] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:43 2022] GET /identity/v3/services?name=compute => generated 124 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:43.282879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:43.348946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:43.388903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4261289-4b97-4b3e-b7bc-2e19d736a4b1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:43.429619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:43.430294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.430580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.430888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.431597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] RBAC: Adding query filter params (type=compute) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:43.431905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:43.433262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8071102-0160-4f6e-b360-c15d59db0f6b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:43.443372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 503/1015] 10.222.0.18 () {56 vars in 1112 bytes} [Sun Jul 24 00:48:43 2022] GET /identity/v3/services?type=compute => generated 352 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:43.446910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:43.547343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07fe237-2b49-4588-971b-13ec5d6ce2bb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:43.581332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:43.581991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.582272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.582546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.583326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:43.585473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-774ae70f-40f7-40c0-ab09-d70d9ff5419a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:43.604919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 513/1016] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:43 2022] POST /identity/v3/endpoints => generated 334 bytes in 160 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:43.754661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c13a37-4af4-4560-8e5c-5509bee2b6d7 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.755182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c13a37-4af4-4560-8e5c-5509bee2b6d7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.755259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4c13a37-4af4-4560-8e5c-5509bee2b6d7 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.757400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 504/1017] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:43.762397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:43.762685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:43.762992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:43.782667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:43.784237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:43.825812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:43.862890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 514/1018] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:43 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:43.888966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c13a37-4af4-4560-8e5c-5509bee2b6d7 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:43.999010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4c13a37-4af4-4560-8e5c-5509bee2b6d7 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:44.028922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:44.037661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:44.038682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:44.038991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:44.039273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:44.040449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:44.041869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57ed914a-bf30-4dc5-8ece-6c3a112b4c82 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:44.043524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 505/1019] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:43 2022] GET /identity/v3/domains/default => generated 199 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:44.049273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:44.153392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c413e44-4b0b-4920-9d77-498e2a230475 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:44.189071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:44.189739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:44.190024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:44.190302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:44.191050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:44.192492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d486e050-ed52-4c33-b279-9c638eb3dc47 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:44.226805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 515/1020] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:48:44 2022] POST /identity/v3/projects => generated 311 bytes in 170 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:44.948275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ba51-fa33-4af5-b958-f91b8c485706 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:44.948803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ba51-fa33-4af5-b958-f91b8c485706 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:44.948803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ca2ba51-fa33-4af5-b958-f91b8c485706 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:44.950327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 506/1021] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:44 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:44.955272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:44.955652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:44.955828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:44.973039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:44.974214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:44.991208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:45.043982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 516/1022] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:44 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:45.066799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2ba51-fa33-4af5-b958-f91b8c485706 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:45.171181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ca2ba51-fa33-4af5-b958-f91b8c485706 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:45.205613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:45.206326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.206614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.206931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] PATH_INFO: `/v3/services/image` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.207608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:45.208936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:45.221196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:45.222189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:45.223498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:45.224597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 507/1023] 10.222.0.18 () {56 vars in 1105 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/services/image => generated 86 bytes in 158 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:45.228547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:45.329930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd1bb96-1541-45bb-a55e-571298190623 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:45.364374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:45.365064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.365887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.366225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.366993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] RBAC: Adding query filter params (name=image) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:45.367305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:45.368730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:45.379016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 517/1024] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/services?name=image => generated 122 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:45.380315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:45.451867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:45.494560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c966f580-d7c6-4612-aee5-ffdb0f15b456 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:45.528488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:45.529157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.529443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.529722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.530427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] RBAC: Adding query filter params (type=image) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:45.530738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:45.532132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:45.542123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 508/1025] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/services?type=image => generated 350 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:45.543175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:45.562419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:45.647368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dfe2a58-307d-443f-bbf7-f80a79f60cf5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:45.681178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:45.681841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.682126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.682401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.683220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] RBAC: Adding query filter params (interface=public, service_id=717affd18fff4a47a893e93b12ab8d28, region_id=RegionOne) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:45.683535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:45.684958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3aec95b4-bd09-4176-a11b-57568fcddebb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:45.690533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 518/1026] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/endpoints?service_id=717affd18fff4a47a893e93b12ab8d28&interface=public®ion_id=RegionOne => generated 194 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:45.696604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:45.769954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338d2f4-f2f5-4800-a1ff-a20967001631 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.770154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338d2f4-f2f5-4800-a1ff-a20967001631 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.770794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f338d2f4-f2f5-4800-a1ff-a20967001631 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.772040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 519/1027] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:45.777157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.777508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.777794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.794679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:45.795900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:45.800945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9962ce6f-a4e3-4975-aa7a-68b515d04994 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:45.826375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:45.838905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:45.839581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:45.839867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:45.840149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:45.841262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:45.843063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:45.858800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 509/1028] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/services => generated 1957 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:48:45.871096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 520/1029] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:45 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:45.893202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.003232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6abe30c-086c-414b-b650-750abfee04b2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.038896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.039644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.039923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.040198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.050497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:46.059517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:46.060723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:46.061941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 510/1030] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:45 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 169 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:46.069241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.081788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.174123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41a4bb7b-c6e6-4aa3-95d1-4b93ce59cdc4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.207749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.208422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.208704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.209531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.210384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:46.210693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.212096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-93978fbb-f002-4670-9d80-abc14a3d7593 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.225470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 521/1031] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:46.227116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.277132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.305570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d672e-2e63-42f6-bb2f-49a0b255f6bd None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.305766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d672e-2e63-42f6-bb2f-49a0b255f6bd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.306052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f1d672e-2e63-42f6-bb2f-49a0b255f6bd None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.307452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 522/1032] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:46.312608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.312942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.313174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.329922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:46.331138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:46.333236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51dfb148-a4d1-4232-963b-c8df7ea1c515 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.367748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.368550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.368829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.369105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.380203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.381812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:46.399803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:46.400730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:46.401635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:46.401635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:46.401635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:46.401635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:46.401635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 511/1033] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/domains/Default => generated 87 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:46.410867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 523/1034] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:46 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:46.411019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.432587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.434840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.511874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.515208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04cd5b0d-432a-4cfa-a22d-79c6d0199a7a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.540847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57f7e7b3-4911-4c07-88a1-337b2b64def3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.551334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.551823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.552112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.552389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.553730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:46.554074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.555439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.568841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 512/1035] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:46.574850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.579954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.580757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.581037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.581311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.582474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.583946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.585714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 524/1036] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/domains/default => generated 199 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:46.591171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.671071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.685873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-204d260c-b5a6-4ed8-b6aa-a104adde418c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.704442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-288bb30c-d8ac-4ca5-b5ac-42e8124616af None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.721250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.722041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.722327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.722594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] PATH_INFO: `/v3/users/nova` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.733252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.734688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.741364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.742034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.742319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.742595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.743337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.744784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:46.749107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:46.750025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 513/1037] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/users/nova => generated 82 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:46.756890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:46.772657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 525/1038] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:48:46 2022] POST /identity/v3/users => generated 285 bytes in 182 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:48:46.783154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:46.872843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b040234d-4cd0-4afa-934b-6f97086d2f5f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:46.911737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:46.912987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:46.913269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:46.913589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:46.914393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:46.914705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:46.916179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:46.935268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 514/1039] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 180 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:46.941516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.052005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a0879e1-fb6a-4f02-8041-3bfd1fcf0f91 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:47.079369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:47.089932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:47.090592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.090901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.091175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.161919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:47.163500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:47.262011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 526/1040] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:46 2022] GET /identity/v3/auth/tokens => generated 1959 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:47.268295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.348284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:47.372374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abdd5f8f-06e1-42f9-ab9e-1b134171673c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:47.409502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:47.410299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.410585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.410893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.424556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:47.426007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:47.445324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:47.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:47.452326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:47.452326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:47.452326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:47.452326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:47.452326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 515/1041] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:47 2022] GET /identity/v3/domains/Default => generated 87 bytes in 179 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:47.452631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.480727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:47.556932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5293d62-3de8-4aa7-8a7d-67e0403c58a5 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:47.592170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:47.592835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.593119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.593397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.594117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:47.594427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:47.595830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:47.611855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 527/1042] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:47 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:47.613164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.716132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-547e9137-7b61-49ba-888b-dcc26fc5ceef None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:47.744109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:47.750736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:47.751559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.751842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.752117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.759359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:47.760831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-89a4edf0-6102-4398-ab57-e6c5f51b7ceb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:47.774317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:47.775438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:47.776368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:47.778844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 516/1043] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:47 2022] GET /identity/v3/projects/service => generated 88 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:47.781287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.871173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:47.887846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-544a87a3-2c1f-47ec-b42e-4bf544840651 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:47.921354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:47.922022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.922307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.922587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.923408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:47.923722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:47.925165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:47.944072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 528/1044] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:47 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 158 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:47.944297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a7a64-6259-4e5b-a38f-3359f2264632 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.944297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a7a64-6259-4e5b-a38f-3359f2264632 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.944297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-192a7a64-6259-4e5b-a38f-3359f2264632 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.945367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:47.945856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 517/1045] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:47 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:47.952806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:47.953174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:47.953533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:47.970657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:47.971899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:48.044531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 518/1046] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:47 2022] POST /identity/v3/auth/tokens => generated 1959 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:48.057199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac5db25-d731-4f5a-a20d-919893de3845 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:48.068190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:48.093296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:48.093955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.094244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.094523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.095376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] RBAC: Adding query filter params (role.id=eb8e58e45a014ce2a398a53f365f1d99, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:48.095691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:48.097138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:48.103648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 529/1047] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:47 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=eb8e58e45a014ce2a398a53f365f1d99 => generated 259 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:48.198546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-93a96dea-1e0b-4aea-a363-82797ce59ad0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:48.219771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:48.236435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:48.237164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.237452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.237732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] PATH_INFO: `/v3/services/image` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.238397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] RBAC: Authorizing `identity:get_service(service_id=image)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:48.239828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] Could not find service: image.: keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:48.252982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:48.254033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: image. Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:48.254926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 519/1048] 10.222.0.18 () {56 vars in 1105 bytes} [Sun Jul 24 00:48:48 2022] GET /identity/v3/services/image => generated 86 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:48.260597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:48.274977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:48.369935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1df0bb61-e919-4ee5-8771-d84a7e28697a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:48.406180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:48.406879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.407166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.407442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.408152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] RBAC: Adding query filter params (name=image) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:48.408458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:48.409833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:48.418297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 530/1049] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:48 2022] GET /identity/v3/services?name=image => generated 122 bytes in 155 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:48.421174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:48.526299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-636336f1-4b03-400c-9119-80e3f19419b2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:48.562184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:48.562880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.563170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.563449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.564155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] RBAC: Adding query filter params (type=image) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:48.564466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:48.565832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be0d33d-8384-4987-a468-687003ec6f3f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:48.576188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 520/1050] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:48 2022] GET /identity/v3/services?type=image => generated 350 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:48.577449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:48.656184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:48.683562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90837329-b52f-4c6a-b645-76d8e3ee4145 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:48.718070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:48.718726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.719034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.719311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] PATH_INFO: `/v3/endpoints` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.720048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:48.721416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ff05b10-08f8-4c1c-b078-768c6041524f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:48.750811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 531/1051] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:48:48 2022] POST /identity/v3/endpoints => generated 327 bytes in 166 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:48:48.889978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe86d0-3afc-46dc-b49e-8d244a3f1c9d None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.890167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe86d0-3afc-46dc-b49e-8d244a3f1c9d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.890443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25fe86d0-3afc-46dc-b49e-8d244a3f1c9d None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.891987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 521/1052] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:48.898435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:48.899508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:48.899508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:48.916833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:48.918020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:48.998826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 532/1053] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:49.018255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe86d0-3afc-46dc-b49e-8d244a3f1c9d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:49.071600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:49.125640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25fe86d0-3afc-46dc-b49e-8d244a3f1c9d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:49.161249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:49.162084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:49.162542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:49.163049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:49.175138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:49.183323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:49.184271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:49.185131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 522/1054] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/roles/admin => generated 83 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:49.192496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:49.253579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:49.300187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51b75c28-ab60-406e-aa67-adfa4801796f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:49.335857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:49.336526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:49.336809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:49.337083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:49.337822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:49.338128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:49.339519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:49.352421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 533/1055] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:49.353607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:49.462628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b17a2032-8948-4d18-80f9-1c720dcbecb5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:49.471951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:49.498438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:49.499333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:49.499613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:49.499885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] PATH_INFO: `/v3/users/fab91afdeba84a3ea688608dc69eea77` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:49.512985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] RBAC: Authorizing `identity:get_user(user_id=fab91afdeba84a3ea688608dc69eea77)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:49.514413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:49.516276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 523/1056] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/users/fab91afdeba84a3ea688608dc69eea77 => generated 285 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:49.521788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:49.621281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26fa8cd9-c9c8-46d3-a2e5-c31d2a0129df None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:49.637833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:49.655388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:49.656034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:49.656313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:49.656587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:49.726188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:49.727576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:49.826457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 534/1057] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:48:49.832077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:49.936173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3758ebaa-9c7b-423b-8a59-cab7767b83c1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:49.970225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:49.971133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:49.971424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:49.971705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:49.972884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:49.974312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e2d9f2-dce7-4d21-8b42-7ee09b6f299d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:49.977197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 524/1058] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:49.983048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.002513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:50.087850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38a9e2ca-8cf4-4530-85c1-ae5e8d5775ae None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:50.121699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:50.122366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.122649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.122960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.123791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9, user.id=fab91afdeba84a3ea688608dc69eea77) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:50.124104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:50.125557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3895cbd5-5bc4-43ce-84a5-d1a0f9dac369 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:50.131624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 535/1059] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:49 2022] GET /identity/v3/role_assignments?user.id=fab91afdeba84a3ea688608dc69eea77&scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:50.159711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f5220-6897-47fe-a458-599ed0d2af02 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.160032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f5220-6897-47fe-a458-599ed0d2af02 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.160279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa7f5220-6897-47fe-a458-599ed0d2af02 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.161936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 525/1060] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:50.168707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.169039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.169267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.188348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:50.189562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:50.268844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 536/1061] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:50.292803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7f5220-6897-47fe-a458-599ed0d2af02 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.329944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:50.403884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa7f5220-6897-47fe-a458-599ed0d2af02 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:50.443974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:50.444796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.445133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.445414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.456290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:50.465747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:50.467151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:50.468012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 526/1062] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 175 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:50.474989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.489399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:50.579013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-695aa2a5-383d-4b76-a4a9-0fbbfd35261c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:50.613341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:50.614020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.614301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.614575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.616174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:50.616477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:50.617914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:50.631336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 537/1063] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:50.632843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.727933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:50.736266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2120dcb-641a-4871-be2d-c50b347c66c7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:50.771544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:50.772536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.772824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.773102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.783977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:50.785419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-665456d3-4bfb-455c-8582-b3949b5dac4a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:50.803444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:50.804384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:50.805274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:50.805274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:50.805274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:50.805274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:50.805274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 527/1064] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:50.811598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.818428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c8304-52f4-4170-a9da-648d522efb72 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.818720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c8304-52f4-4170-a9da-648d522efb72 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.819036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c5c8304-52f4-4170-a9da-648d522efb72 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.820623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 528/1065] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:50.825857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.826273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.826541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.844810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:50.845908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:50.893648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:50.920829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 529/1066] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:50 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:50.923115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f681548-a931-42e2-a1f0-e400e599e699 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:50.942729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:50.960543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:50.961217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:50.961506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:50.961785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:50.962506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:50.962836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:50.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:50.981312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 538/1067] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 166 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:50.982940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.031766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.056034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08a4231f-c8d4-4362-8d39-05c53389b44c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.086651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3faace49-975c-485f-adeb-30cf3a415a5e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.092032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.092682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.092965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.093238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.123356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.124208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.124497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.124776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.126841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.135190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.137250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:51.154674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:51.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 539/1068] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/users/nova => generated 82 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:51.162451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.170987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.172310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.253587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.276644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb5bd8e-9300-496b-86b1-94be56a7b465 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.278300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 530/1069] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:50 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:51.283842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.318926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.319897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.319897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.320170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.320971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:51.321281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.323194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.342017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 540/1070] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 181 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:51.347319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.393184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b16a7d4-10b7-4421-8c3e-bb6331fef173 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.407418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.433851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.434657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.434972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.435250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.446630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.448088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.457780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-67f4cee2-159a-4b2c-a5a0-bd8a7bb79739 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:51.466560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:51.467825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:51.468750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:51.468750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:51.468750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:51.468750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:51.468750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 531/1071] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 184 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:51.473727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.500474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.501369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.501652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.501924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.502620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.512003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.513449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:51.531711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:51.532656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:51.533561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:51.533561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:51.533561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:51.533561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:51.533561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 541/1072] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/domains/Default => generated 87 bytes in 186 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:51.539136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.589054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d401066-095b-4104-9ea4-3bc390cc2b8c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.631213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.631869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.632158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.632439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.633163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:51.633476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.634948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.642491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee630634-5d6c-4aeb-b301-5120e36836c0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.645633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.647613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 532/1073] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:51.689347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.690032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.690317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.690591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.691371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:51.691681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.693082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.712576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 542/1074] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:51.713717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:51.814286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:51.826846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92844018-79b5-4665-8126-3ab0f58d1230 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:51.864473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:51.865275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:51.865562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:51.865890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:51.873014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:51.874441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:51.889220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:51.890211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:51.891519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 533/1075] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:51.896267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.002306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b97ef24-4217-4307-8eaf-73d357e826a0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:52.035868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:52.036528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.036810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.037289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.038088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:52.038395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:52.039868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2299b40d-e3bb-4dc7-9abb-41656509f428 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:52.051784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 543/1076] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:51 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:52.057822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.155339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b0a20a-f344-4cba-bde9-d65c88451bfb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:52.183429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4c45e-c33f-4134-9212-e6071185cc48 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.183729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4c45e-c33f-4134-9212-e6071185cc48 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.184012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a4c45e-c33f-4134-9212-e6071185cc48 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.185375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 544/1077] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:52 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:52.189309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:52.190189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.190466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.190559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.190736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/eb8e58e45a014ce2a398a53f365f1d99` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.190851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.191090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.192923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=865a61591b7d45ca90431a7683e96a61, role_id=eb8e58e45a014ce2a398a53f365f1d99)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:52.195138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:52.207864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 534/1078] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:52 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/865a61591b7d45ca90431a7683e96a61/roles/eb8e58e45a014ce2a398a53f365f1d99 => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:52.209527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:52.210733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:52.230434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:52.288195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 545/1079] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:52 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:52.311902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.324403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:52.431139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-871bc449-9c50-4d01-85a9-a8fbc8b8b1f7 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:52.476096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:52.476900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.477183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.477454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.489701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:52.499870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:52.500819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:52.501661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 535/1080] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:52 2022] GET /identity/v3/roles/admin => generated 83 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:52.508083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.615567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4143d4fe-5e8f-4f5d-b1e8-ccb941d44f91 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:52.651423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:52.652093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.652375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.652647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.653394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:52.653700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:52.655097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:52.664121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 546/1081] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:52 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:52.670355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.751743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:52.788881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5e892a-827d-4318-8796-21b7d81f2ca8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:52.825805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:52.826923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:52.827204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:52.827475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] PATH_INFO: `/v3/users/fab91afdeba84a3ea688608dc69eea77` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:52.828663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] RBAC: Authorizing `identity:get_user(user_id=fab91afdeba84a3ea688608dc69eea77)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:52.830230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:52.837393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 536/1082] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:48:52 2022] GET /identity/v3/users/fab91afdeba84a3ea688608dc69eea77 => generated 285 bytes in 162 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:52.840598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:52.867108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:52.961120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa81ad8-e400-45b2-9f3d-0cbc43410620 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:53.007155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:53.008662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:53.009131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:53.009418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:53.010830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:53.012302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d1a1943-6070-4ca6-9bf4-8ef31d06ce97 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:53.014003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 547/1083] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:48:52 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:53.019940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:53.118695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27f69bc6-b5c6-43ea-a91a-95fdb8bf8632 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:53.151950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:53.152837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:53.153119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:53.153392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9/users/fab91afdeba84a3ea688608dc69eea77/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:53.155376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] RBAC: Authorizing `identity:create_grant(project_id=6bd5a3550ccf4bfdac0b275092ce23f9, user_id=fab91afdeba84a3ea688608dc69eea77, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:53.156799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92cd6916-d7c5-46cd-b354-d525758f676f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:53.159776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:53.171565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 537/1084] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:48:53 2022] PUT /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9/users/fab91afdeba84a3ea688608dc69eea77/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:48:54.322130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df949a0f-2a86-4668-811d-6775b4826bb2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:54.322662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df949a0f-2a86-4668-811d-6775b4826bb2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:54.322662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df949a0f-2a86-4668-811d-6775b4826bb2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:54.324139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 548/1085] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:54.329374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:54.329784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:54.329937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:54.346925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:54.348100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:54.417871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 538/1086] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:54 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:54.441793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df949a0f-2a86-4668-811d-6775b4826bb2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:54.524975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:54.542333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df949a0f-2a86-4668-811d-6775b4826bb2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:54.576641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:54.577388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:54.577665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:54.577937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] PATH_INFO: `/v3/roles/ResellerAdmin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:54.587926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] Could not find role: ResellerAdmin.: keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:54.596214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:54.597336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: ResellerAdmin. Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:54.598231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 549/1087] 10.222.0.18 () {56 vars in 1120 bytes} [Sun Jul 24 00:48:54 2022] GET /identity/v3/roles/ResellerAdmin => generated 91 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:54.603518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:54.700788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fd37f7-6a02-41f9-acc2-43887ff044cf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:54.725978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:54.734456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:54.735164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:54.735455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:54.735731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:54.736478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] RBAC: Adding query filter params (name=ResellerAdmin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:54.736787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:54.738148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:54.746835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 539/1088] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:48:54 2022] GET /identity/v3/roles?name=ResellerAdmin => generated 338 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:54.752423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:54.851181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd80b6fc-6df0-476c-b108-34fe75179392 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:54.885066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:54.885858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:54.886142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:54.886415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:54.897085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:54.898536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:54.901018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:54.917289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:54.918186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:54.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:54.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:54.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:54.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:54.919119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 550/1089] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 167 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:54.924168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.022053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50fac94e-9adc-47ae-9d6c-d97da2a8ddce None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.056312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.056978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.057265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.057544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.058263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:55.058574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.059972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf5e5f33-1a28-4f9a-9c37-6b82d7fb0e86 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.072092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 540/1090] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 148 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:55.077643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.175589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108db70-d409-49df-bb35-8d7679c3238a None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.175877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108db70-d409-49df-bb35-8d7679c3238a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.176159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4108db70-d409-49df-bb35-8d7679c3238a None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.177585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 541/1091] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:55 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:48:55.179056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-198da5fc-23e2-4e91-8b31-6d3ad41421b1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.182525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.182836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.183122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.199626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:55.200838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:55.212607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.213395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.213673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.213945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] PATH_INFO: `/v3/users/nova` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.221287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] RBAC: Authorizing `identity:get_user(user_id=nova)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.222744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] Could not find user: nova.: keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:48:55.237359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:55.238318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: nova. Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:55.239286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 551/1092] 10.222.0.18 () {56 vars in 1093 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/users/nova => generated 82 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:48:55.244176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.245025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.268634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 542/1093] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:55 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:48:55.292645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.299454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.345611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd5808fa-7fcb-40d2-8b07-6cc1077e53e0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.378340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.379024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.379305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.379580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.380395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] RBAC: Adding query filter params (domain_id=default, name=nova) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:55.380829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.382266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.389971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de3cf95c-85b3-4473-9cd9-94245efde1a1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.400058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 552/1094] 10.222.0.18 () {56 vars in 1133 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/users?domain_id=default&name=nova => generated 368 bytes in 156 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:55.405615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.433751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.434566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.434885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.435163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.446098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:48:55.453787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:55.454931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:48:55.455798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 543/1095] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/roles/admin => generated 83 bytes in 163 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:55.461627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.499627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.504205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4bddff5-4d3c-438d-a27b-287664e3a02d None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.518326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.539477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.540129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.540415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.540692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.560069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39bcfc5e-aba6-4ee5-86e9-03d4f02de3ea None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.594160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.594856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.595147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.595435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.596176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:55.596522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.597885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.606703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 544/1096] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:55.607615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.609069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.612409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.707483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.709899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 553/1097] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:48:55.714710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-541d5ed4-8b42-4aad-9bb0-76336d553775 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.715602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.729814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.745531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.746398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.746677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.746981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] PATH_INFO: `/v3/users/fab91afdeba84a3ea688608dc69eea77` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.748166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] RBAC: Authorizing `identity:get_user(user_id=fab91afdeba84a3ea688608dc69eea77)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.749578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.751227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 545/1098] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/users/fab91afdeba84a3ea688608dc69eea77 => generated 285 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:55.757399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.812932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d5ebb9-7e7b-4328-ad14-3d736ad242bb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.847159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.847943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.848223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.848494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.853308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cad4506-14f3-4a0a-bc4a-5bfe18d76ee0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:55.858265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.859756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:48:55.877627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:55.878811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:55.879730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:48:55.879730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:48:55.879730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:48:55.879730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:55.879730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 554/1099] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/domains/Default => generated 87 bytes in 164 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:55.885126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:55.888732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:55.889384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:55.889673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:55.889952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:55.923370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.949878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:55.958757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:55.961028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:55.986311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6456f7fe-3bf6-4e60-918e-ff4ce9ee55d9 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.020743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.021459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.021743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.022019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.022733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:56.023071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.024432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.035647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 555/1100] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:56.041357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:56.058963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 546/1101] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:48:55 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 302 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:48:56.064974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:56.128622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:56.140399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9510b72-993e-4cc0-9a0c-c2d652d7dccf None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.161926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb25f09-8db6-49e8-bc7e-de4ee3de5d42 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.165009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:56.174630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.175448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.175723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.176008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.183257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.184692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.196613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.197468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.197756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.198035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.199235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:48:56.199379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:48:56.200412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:48:56.201356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:48:56.201787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.201856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 556/1102] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:48:56 2022] GET /identity/v3/projects/service => generated 88 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:48:56.202798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 547/1103] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:48:56 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:56.206281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:56.208219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:56.303874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f37db008-398a-4a21-9370-70e3beb1c727 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.306242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bdf0687-3e87-4d7c-a500-76692e20d6fb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.337808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.338456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.338740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.339038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.339580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.339806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:56.340110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.340210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.340495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.340774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.341524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5483d019-0e5e-434b-976d-353c1e0ce4db None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.341633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9, user.id=fab91afdeba84a3ea688608dc69eea77) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:56.341893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.343338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.346106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:56.350444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 548/1104] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:56 2022] GET /identity/v3/role_assignments?user.id=fab91afdeba84a3ea688608dc69eea77&scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 142 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:48:56.356552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 557/1105] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:48:56 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:56.362893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:56.470872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60dbf146-8033-4808-9c03-602e544fa138 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:56.506441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:56.507131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:56.507410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:56.507684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:56.508502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] RBAC: Adding query filter params (role.id=eb8e58e45a014ce2a398a53f365f1d99, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=865a61591b7d45ca90431a7683e96a61) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:48:56.508814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:56.510235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062b4f7f-9714-4fc0-8562-cb4370db0072 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:56.516311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 549/1106] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:48:56 2022] GET /identity/v3/role_assignments?user.id=865a61591b7d45ca90431a7683e96a61&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=eb8e58e45a014ce2a398a53f365f1d99 => generated 608 bytes in 154 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:58.401437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc2f5-8d64-4389-8ece-5875685e7ca3 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:58.401969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc2f5-8d64-4389-8ece-5875685e7ca3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:58.402061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2dbdc2f5-8d64-4389-8ece-5875685e7ca3 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:58.403398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 558/1107] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:48:58 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:48:58.408722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:58.409139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:58.409285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:58.426062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:48:58.427825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:48:58.497717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:58.506186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 550/1108] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:48:58 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:48:58.530408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbdc2f5-8d64-4389-8ece-5875685e7ca3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:58.578201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:48:58.631777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2dbdc2f5-8d64-4389-8ece-5875685e7ca3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:58.664573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:58.665354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:58.665640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:58.665918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:58.667127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:58.668562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff645215-12a6-4dff-93dc-47d8da60bda5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:58.670146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 559/1109] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:48:58 2022] GET /identity/v3/domains/default => generated 199 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:48:58.675795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:48:58.772476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-27034820-4bbb-4c29-ba0c-1b9179d3328a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:48:58.805843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:48:58.806507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:48:58.806819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:48:58.807104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:48:58.807818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:48:58.809243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41bdf7e-71c7-43de-84f5-d130e9655552 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:48:58.828495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 551/1110] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:48:58 2022] POST /identity/v3/users => generated 286 bytes in 153 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:49:00.909743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664a627a-edaa-427a-8897-72680dd4274c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:00.910299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664a627a-edaa-427a-8897-72680dd4274c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:00.910299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-664a627a-edaa-427a-8897-72680dd4274c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:00.912296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 560/1111] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:00.917518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:00.917939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:00.918064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:00.935201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:00.936635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:00.981654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:01.003176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 552/1112] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:00 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:01.026993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664a627a-edaa-427a-8897-72680dd4274c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:01.127733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-664a627a-edaa-427a-8897-72680dd4274c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:01.141250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:01.161503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:01.162958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:01.163239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:01.163536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] PATH_INFO: `/v3/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:01.164991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] RBAC: Authorizing `identity:get_role(role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:01.166363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:01.167929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 561/1113] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:01 2022] GET /identity/v3/roles/496aed2e00104d4480f0efe375a79409 => generated 223 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:01.173601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:01.268172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7de9691-ddab-48c3-80e5-cf92a5fa6b3a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:01.300706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:01.301888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:01.302235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:01.302509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] PATH_INFO: `/v3/users/dd639539d95847abb5ff643b4c9ce0ae` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:01.315152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] RBAC: Authorizing `identity:get_user(user_id=dd639539d95847abb5ff643b4c9ce0ae)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:01.316574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:01.318199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 553/1114] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:01 2022] GET /identity/v3/users/dd639539d95847abb5ff643b4c9ce0ae => generated 286 bytes in 145 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:01.323843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:01.420258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a800e284-c30e-444b-981b-cf4efbd50f6e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:01.453541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:01.454178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:01.454461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:01.454737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:01.501821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:01.519908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:01.521487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:01.616310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 562/1115] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:01 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:01.622149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:01.659901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:01.717244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19f31aca-8128-4359-894a-c79c2c5704db None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:01.749623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:01.750469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:01.750752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:01.751056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:01.753333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:01.754794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e058a0-69b9-47cb-966f-ccfbc7062914 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:01.756511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 554/1116] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:01 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:01.762498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:01.841672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:01.859943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b042a23-01f9-4c14-a254-c6395041e26a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:01.891854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:01.892491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:01.892776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:01.893052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:01.893868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9, user.id=dd639539d95847abb5ff643b4c9ce0ae) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:01.894178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:01.895630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a5c89dd-a642-4c78-9d39-f3d352e8bba4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:01.901631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 563/1117] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:01 2022] GET /identity/v3/role_assignments?user.id=dd639539d95847abb5ff643b4c9ce0ae&scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9&role.id=496aed2e00104d4480f0efe375a79409 => generated 259 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:03.873179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bb084-efb3-4e93-a28a-45d649465ace None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:03.873727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bb084-efb3-4e93-a28a-45d649465ace None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:03.873727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc9bb084-efb3-4e93-a28a-45d649465ace None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:03.875052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 555/1118] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:03 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:03.880286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:03.880720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:03.880846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:03.897117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:03.898300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:03.963815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 564/1119] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:03 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:03.986722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9bb084-efb3-4e93-a28a-45d649465ace None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:04.085659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc9bb084-efb3-4e93-a28a-45d649465ace None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:04.092533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:04.118872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:04.119745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:04.120025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:04.120298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] PATH_INFO: `/v3/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:04.121741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] RBAC: Authorizing `identity:get_role(role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:04.123125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:04.124534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 556/1120] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:03 2022] GET /identity/v3/roles/496aed2e00104d4480f0efe375a79409 => generated 223 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:04.130279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:04.203749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:04.227364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1007b568-01ef-48a8-9ed5-ffd472940c8d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:04.260211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:04.261067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:04.261347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:04.261619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] PATH_INFO: `/v3/users/dd639539d95847abb5ff643b4c9ce0ae` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:04.262811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] RBAC: Authorizing `identity:get_user(user_id=dd639539d95847abb5ff643b4c9ce0ae)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:04.264239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:04.265840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 565/1121] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:04 2022] GET /identity/v3/users/dd639539d95847abb5ff643b4c9ce0ae => generated 286 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:04.272452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:04.368452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceabc8a4-3131-44fd-973c-277f921828d8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:04.401085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:04.401867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:04.402148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:04.402423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:04.403602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:04.405020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3376fce3-8e90-463b-b1d0-43ce30a1f543 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:04.406687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 557/1122] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:04 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:04.413556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:04.509117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a99c96ae-8906-47d3-88f2-f172017ee099 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:04.541713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:04.542586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:04.542895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:04.543172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9/users/dd639539d95847abb5ff643b4c9ce0ae/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:04.545320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] RBAC: Authorizing `identity:create_grant(project_id=6bd5a3550ccf4bfdac0b275092ce23f9, user_id=dd639539d95847abb5ff643b4c9ce0ae, role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:04.546747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30874c53-e88e-4594-b008-98bfe964d97e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:04.559221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 566/1123] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:49:04 2022] PUT /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9/users/dd639539d95847abb5ff643b4c9ce0ae/roles/496aed2e00104d4480f0efe375a79409 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:49:06.603633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f177472-d991-4ad3-9c3a-11c5400d62e5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:06.604177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f177472-d991-4ad3-9c3a-11c5400d62e5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:06.604177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f177472-d991-4ad3-9c3a-11c5400d62e5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:06.605555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 558/1124] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:06 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:06.610543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:06.610991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:06.611127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:06.618088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:06.629434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:06.631760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:06.698885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 567/1125] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:06 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:06.722357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f177472-d991-4ad3-9c3a-11c5400d62e5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:06.765343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:06.820581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f177472-d991-4ad3-9c3a-11c5400d62e5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:06.852619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:06.853458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:06.853738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:06.854011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] PATH_INFO: `/v3/roles/496aed2e00104d4480f0efe375a79409` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:06.855487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] RBAC: Authorizing `identity:get_role(role_id=496aed2e00104d4480f0efe375a79409)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:06.856831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:06.858181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 559/1126] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:06 2022] GET /identity/v3/roles/496aed2e00104d4480f0efe375a79409 => generated 223 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:06.863951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:06.961032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-411a4ea9-d615-4e29-b9f4-b859248dd0fd None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:06.994960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:06.995835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:06.996120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:06.996397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] PATH_INFO: `/v3/users/dd639539d95847abb5ff643b4c9ce0ae` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:06.997577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] RBAC: Authorizing `identity:get_user(user_id=dd639539d95847abb5ff643b4c9ce0ae)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:06.999121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:07.000738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 568/1127] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:06 2022] GET /identity/v3/users/dd639539d95847abb5ff643b4c9ce0ae => generated 286 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:07.006656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:07.101805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ba3321f-05d5-41eb-914b-0d778339cb82 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:07.111976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:07.134701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:07.135386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:07.135679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:07.135954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:07.200751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:07.202199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:07.296408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 560/1128] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:07 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 290 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:07.302436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:07.305309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:07.400309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d70298-d1dd-4233-b003-2dc4ff3f6692 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:07.433603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:07.434463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:07.434747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:07.435052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] PATH_INFO: `/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:07.436482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] RBAC: Authorizing `identity:get_project(project_id=6bd5a3550ccf4bfdac0b275092ce23f9)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:07.437921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9918d4d4-b017-46f9-a7ac-a6d933ea722b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:07.439645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 569/1129] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:07 2022] GET /identity/v3/projects/6bd5a3550ccf4bfdac0b275092ce23f9 => generated 311 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:07.446210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:07.457492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:07.542921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1c34b7f-cca5-4c4d-87ea-565989331ed0 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:07.576863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:07.577445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:07.577727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:07.578005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:07.578846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] RBAC: Adding query filter params (role.id=496aed2e00104d4480f0efe375a79409, scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9, user.id=dd639539d95847abb5ff643b4c9ce0ae) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:07.579162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:07.580585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01ea8149-87c6-4fab-9ac7-ed4b3c80c455 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:07.587101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 561/1130] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:07 2022] GET /identity/v3/role_assignments?user.id=dd639539d95847abb5ff643b4c9ce0ae&scope.project.id=6bd5a3550ccf4bfdac0b275092ce23f9&role.id=496aed2e00104d4480f0efe375a79409 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:09.600872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558872d-f586-4f4b-a3d1-61460f26b3fc None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:09.601409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558872d-f586-4f4b-a3d1-61460f26b3fc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:09.601409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2558872d-f586-4f4b-a3d1-61460f26b3fc None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:09.602756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 570/1131] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:09.607909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:09.608347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:09.608473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:09.625217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:09.626384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:09.690457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 562/1132] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:09 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:49:09.713509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558872d-f586-4f4b-a3d1-61460f26b3fc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:09.781416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:09.813264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2558872d-f586-4f4b-a3d1-61460f26b3fc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:09.845901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:09.846676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:09.846982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:09.847255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:09.848413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:09.850426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef16ad36-ac37-44a6-9166-0838b8bed29a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:09.852038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 571/1133] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:49:09 2022] GET /identity/v3/domains/default => generated 199 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:09.858030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:09.861080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:09.953580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50b0ed3e-cd35-40bf-b4ca-d3339343cde1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:09.986536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:09.987207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:09.987488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:09.987759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:09.988461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:09.990950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-592eec4e-afe4-4d0e-ba18-bf3c0dacf382 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:10.015874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 563/1134] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:49:09 2022] POST /identity/v3/projects => generated 311 bytes in 158 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:49:12.049908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a69e8-c0c3-4c57-859f-8961dae65c27 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:12.050470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a69e8-c0c3-4c57-859f-8961dae65c27 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:12.050470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d0a69e8-c0c3-4c57-859f-8961dae65c27 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:12.052022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 572/1135] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:12.057093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:12.057493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:12.057661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:12.074812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:12.075995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:12.123327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:12.142491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 564/1136] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:12 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:12.168760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0a69e8-c0c3-4c57-859f-8961dae65c27 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:12.267710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d0a69e8-c0c3-4c57-859f-8961dae65c27 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:12.301125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:12.301881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:12.302159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:12.302430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] PATH_INFO: `/v3/domains/default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:12.303632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] RBAC: Authorizing `identity:get_domain(domain_id=default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:12.305072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce97bb3d-ec67-4d5d-a7ec-4c3c08de0bb4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:12.306659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 573/1137] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:49:12 2022] GET /identity/v3/domains/default => generated 199 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:12.312521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:12.406991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0167948-164a-4de7-9c7f-0ed8b44dddce None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:12.440275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:12.440940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:12.441225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:12.441505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:12.443015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:12.444451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e46093ba-f40c-4089-b028-d8138d50f02a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:12.463598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 565/1138] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:49:12 2022] POST /identity/v3/users => generated 286 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:49:14.504572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0eb165-974e-4d5f-8c35-6989eee1b142 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:14.505491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0eb165-974e-4d5f-8c35-6989eee1b142 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:14.505491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c0eb165-974e-4d5f-8c35-6989eee1b142 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:14.506649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 574/1139] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:14 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:14.511600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:14.512021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:14.512159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:14.528808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:14.530058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:14.565310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:14.596660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 566/1140] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:14 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:14.620490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0eb165-974e-4d5f-8c35-6989eee1b142 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:14.629190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:14.721736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c0eb165-974e-4d5f-8c35-6989eee1b142 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:14.755322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:14.756104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:14.756389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:14.756663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:14.766475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:14.774399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:14.775635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:14.776592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 575/1141] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:14 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:14.781519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:14.876413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18dc73f8-43d2-40f2-b293-4df85ae9f479 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:14.908975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:14.909624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:14.909909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:14.910184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:14.911523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:14.912055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:14.913413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:14.921190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 567/1142] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:14 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:14.927499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:14.991864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:15.026694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5199f9-43a1-4e7c-af5a-76650463ffa3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:15.059642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:15.060773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:15.061053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:15.061326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] PATH_INFO: `/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:15.074251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] RBAC: Authorizing `identity:get_user(user_id=74c43fce599f4dd9a76f941a4c7d5ccf)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:15.075725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:15.077388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 576/1143] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:14 2022] GET /identity/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf => generated 286 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:15.083670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:15.117389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:15.182718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5a9013-64e3-40f5-b647-42d94b8a9824 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:15.215115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:15.215770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:15.216041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:15.216317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:15.280553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:15.281924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:15.313457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:15.377396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 568/1144] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:15 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:15.383828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:15.482174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb4ef9b-3eb4-4796-a406-4c7f4a80ace7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:15.504435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:15.515671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:15.516521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:15.516801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:15.517072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] PATH_INFO: `/v3/projects/65ff5777a636444595f3dee06e851967` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:15.518238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] RBAC: Authorizing `identity:get_project(project_id=65ff5777a636444595f3dee06e851967)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:15.519696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159f307e-c32a-4922-8eab-037900d4f450 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:15.521963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 577/1145] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:15 2022] GET /identity/v3/projects/65ff5777a636444595f3dee06e851967 => generated 311 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:15.528068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:15.623814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-813f6a9c-4275-4413-b57c-90b456000394 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:15.657342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:15.657971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:15.658252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:15.658528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:15.659377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=65ff5777a636444595f3dee06e851967, user.id=74c43fce599f4dd9a76f941a4c7d5ccf) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:15.659687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:15.662148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4346bf37-9cb7-4399-a727-825feac20dab None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:15.663944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:15.674822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 569/1146] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:15 2022] GET /identity/v3/role_assignments?user.id=74c43fce599f4dd9a76f941a4c7d5ccf&scope.project.id=65ff5777a636444595f3dee06e851967&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:17.675411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552a557-4c75-4c3f-8430-1f5bd9ba0a75 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:17.675958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552a557-4c75-4c3f-8430-1f5bd9ba0a75 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:17.675958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4552a557-4c75-4c3f-8430-1f5bd9ba0a75 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:17.677843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 578/1147] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:17.682977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:17.683397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:17.683540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:17.700385 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:17.701546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:17.766314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 570/1148] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:17 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:17.789217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552a557-4c75-4c3f-8430-1f5bd9ba0a75 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:17.887100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4552a557-4c75-4c3f-8430-1f5bd9ba0a75 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:17.919604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:17.920378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:17.920654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:17.920920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:17.931338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:17.939163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:17.940280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:17.941347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 579/1149] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:17 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:17.946372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:18.042861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cccbba5-a428-4581-97ef-4402ad88c6bf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:18.068690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:18.076755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:18.077419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:18.077708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:18.077988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:18.078731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:18.079073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:18.080435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:18.089017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 571/1150] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:17 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:18.094817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:18.146183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:18.191260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc1192ca-4f13-4d15-ab9a-8dce21623262 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:18.223035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:18.224695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:18.224972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:18.225241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] PATH_INFO: `/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:18.226400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] RBAC: Authorizing `identity:get_user(user_id=74c43fce599f4dd9a76f941a4c7d5ccf)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:18.227856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:18.229444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 580/1151] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:18 2022] GET /identity/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf => generated 286 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:18.235233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:18.331214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ec6286e-52f4-415d-8df1-fa0b9593b4d2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:18.364011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:18.364849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:18.365137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:18.365414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] PATH_INFO: `/v3/projects/65ff5777a636444595f3dee06e851967` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:18.366585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] RBAC: Authorizing `identity:get_project(project_id=65ff5777a636444595f3dee06e851967)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:18.368043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-614443bb-423a-4335-9181-0291983ca1d1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:18.369708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 572/1152] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:18 2022] GET /identity/v3/projects/65ff5777a636444595f3dee06e851967 => generated 311 bytes in 135 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:18.375486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:18.476193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d5bb773-a9b7-4058-97a8-c6ca39051113 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:18.495085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:18.510075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:18.510996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:18.511282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:18.511559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] PATH_INFO: `/v3/projects/65ff5777a636444595f3dee06e851967/users/74c43fce599f4dd9a76f941a4c7d5ccf/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:18.513557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] RBAC: Authorizing `identity:create_grant(project_id=65ff5777a636444595f3dee06e851967, user_id=74c43fce599f4dd9a76f941a4c7d5ccf, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:18.515025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31400d2c-7c42-4c8c-a677-508c09abbef1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:18.528626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 581/1153] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:49:18 2022] PUT /identity/v3/projects/65ff5777a636444595f3dee06e851967/users/74c43fce599f4dd9a76f941a4c7d5ccf/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:49:20.512270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf71d4-ec4e-4113-8445-39668f3312f8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:20.512821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf71d4-ec4e-4113-8445-39668f3312f8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:20.512821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20bf71d4-ec4e-4113-8445-39668f3312f8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:20.514185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 573/1154] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:20.519272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:20.519689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:20.519834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:20.537722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:20.538938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:20.604058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 582/1155] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:20 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:49:20.626992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf71d4-ec4e-4113-8445-39668f3312f8 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:20.684479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:20.726077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20bf71d4-ec4e-4113-8445-39668f3312f8 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:20.757833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:20.758606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:20.758922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:20.759200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:20.769305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:20.777566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:20.778713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:49:20.779953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 574/1156] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:20 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:20.784666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:20.879827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f7e49b0-dae3-4810-a438-b153351472f3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:20.890050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:20.914180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:20.914874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:20.915159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:20.915432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:20.916176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:20.916476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:20.917836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:20.925609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 583/1157] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:20 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:20.931358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:21.027953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-938e0bbc-773b-4612-aafb-60528f73138b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:21.045276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:21.062639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:21.063541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:21.063829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:21.064111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] PATH_INFO: `/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:21.065283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] RBAC: Authorizing `identity:get_user(user_id=74c43fce599f4dd9a76f941a4c7d5ccf)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:21.066825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:21.069118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 575/1158] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:20 2022] GET /identity/v3/users/74c43fce599f4dd9a76f941a4c7d5ccf => generated 286 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:21.074813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:21.171918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64d8ad9a-92c4-48b4-b9d7-58fbcd9389a7 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:21.205001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:21.205635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:21.205925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:21.206204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:21.245720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:21.272062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:21.273534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:21.367720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 584/1159] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:21 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 293 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:21.373473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:21.474552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceade5b3-2803-4678-85af-83e32b9578c4 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:21.507454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:21.508265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:21.508549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:21.508827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] PATH_INFO: `/v3/projects/65ff5777a636444595f3dee06e851967` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:21.509997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] RBAC: Authorizing `identity:get_project(project_id=65ff5777a636444595f3dee06e851967)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:21.511460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be42d56a-7e19-47f4-863e-fe731790e688 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:21.513143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 576/1160] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:21 2022] GET /identity/v3/projects/65ff5777a636444595f3dee06e851967 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:21.519213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:21.588218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:21.617427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-687cdbb0-e7fa-4b8e-8ed4-82791bf26fef None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:21.650452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:21.651067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:21.651351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:21.651628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:21.652446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=65ff5777a636444595f3dee06e851967, user.id=74c43fce599f4dd9a76f941a4c7d5ccf) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:21.652759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:21.654185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74125cb8-52fd-4396-97d5-23e8cc8236c5 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:21.659871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 585/1161] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:21 2022] GET /identity/v3/role_assignments?user.id=74c43fce599f4dd9a76f941a4c7d5ccf&scope.project.id=65ff5777a636444595f3dee06e851967&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:23.701491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841644e6-f7e0-46a9-ad62-76179c99d5a8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:23.701980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841644e6-f7e0-46a9-ad62-76179c99d5a8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:23.701980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-841644e6-f7e0-46a9-ad62-76179c99d5a8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:23.703379 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 577/1162] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:23.708481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:23.708923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:23.709025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:23.725924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:23.727150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:23.791748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 586/1163] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:23 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:23.814184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841644e6-f7e0-46a9-ad62-76179c99d5a8 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:23.868941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:23.912286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-841644e6-f7e0-46a9-ad62-76179c99d5a8 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:23.945849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:23.946486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:23.946796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:23.947078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:24.012320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:24.013694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:24.065038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:24.108041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 578/1164] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:23 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 294 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:24.114607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:24.177644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:24.211835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fb372e1-10e8-470b-bc7c-ecd51b010e0e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:24.244427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:24.245223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:24.245506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:24.245783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] PATH_INFO: `/v3/domains/swift_test` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:24.252298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] RBAC: Authorizing `identity:get_domain(domain_id=swift_test)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:24.253733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-50130fea-0c82-4658-9198-48f80889ee65 None admin] Could not find domain: swift_test.: keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 712, in get_domain Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = self.driver.get_project(domain_id) Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: swift_test. Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:24.268220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:24.269443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 718, in get_domain Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: swift_test. Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:24.271138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 587/1165] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:49:24 2022] GET /identity/v3/domains/swift_test => generated 90 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:24.275478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:24.371059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3bc6976-b011-4d63-b64e-697affa71fc1 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:24.403853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:24.404506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:24.404791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:24.405069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:24.405781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] RBAC: Adding query filter params (name=swift_test) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:24.406089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:24.407482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7944a6d6-e69b-4d19-b299-45f494322c0d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:24.415149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 579/1166] 10.222.0.18 () {56 vars in 1115 bytes} [Sun Jul 24 00:49:24 2022] GET /identity/v3/domains?name=swift_test => generated 125 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:26.434015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d67fa-2a6f-41b6-a2d2-e9c56530daa6 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:26.434422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d67fa-2a6f-41b6-a2d2-e9c56530daa6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:26.434561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab4d67fa-2a6f-41b6-a2d2-e9c56530daa6 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:26.436228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 588/1167] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:26.441478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888f8c1-0a93-4379-8b6f-b07410a45085 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:26.441947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888f8c1-0a93-4379-8b6f-b07410a45085 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:26.442237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5888f8c1-0a93-4379-8b6f-b07410a45085 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:26.459107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5888f8c1-0a93-4379-8b6f-b07410a45085 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:26.460294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5888f8c1-0a93-4379-8b6f-b07410a45085 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:26.473521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:26.525754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 580/1168] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:26 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:26.549283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4d67fa-2a6f-41b6-a2d2-e9c56530daa6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:26.650895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab4d67fa-2a6f-41b6-a2d2-e9c56530daa6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:26.676794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:26.685721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:26.686360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:26.686638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:26.686942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:26.688342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] RBAC: Authorizing `identity:create_domain()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:26.689771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-24e781c7-c0b8-4fa7-849d-c1c07547adc1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:26.704226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 589/1169] 10.222.0.18 () {60 vars in 1138 bytes} [Sun Jul 24 00:49:26 2022] POST /identity/v3/domains => generated 267 bytes in 155 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 00:49:28.748186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f85adaf-3a17-412a-802a-7c7caff06d88 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:28.748691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f85adaf-3a17-412a-802a-7c7caff06d88 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:28.748691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f85adaf-3a17-412a-802a-7c7caff06d88 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:28.750044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 581/1170] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:28 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:28.755918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:28.756332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:28.756468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:28.772785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:28.773963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:28.838147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 590/1171] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:28 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:28.861205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f85adaf-3a17-412a-802a-7c7caff06d88 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:28.959723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f85adaf-3a17-412a-802a-7c7caff06d88 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:28.993344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:28.994202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:28.994492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:28.994795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] PATH_INFO: `/v3/domains/53a0948fd9ca455692470d4878e97fa1` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:29.005077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] RBAC: Authorizing `identity:get_domain(domain_id=53a0948fd9ca455692470d4878e97fa1)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:29.006461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ec9acad-8a5b-499e-a7d1-3527d6b18441 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:29.008126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 582/1172] 10.222.0.18 () {56 vars in 1183 bytes} [Sun Jul 24 00:49:28 2022] GET /identity/v3/domains/53a0948fd9ca455692470d4878e97fa1 => generated 267 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:29.013966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:29.110945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ec4db8b-a0bd-426d-b28a-76100fb1c9fb None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:29.127403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:29.145460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:29.146106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:29.146385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:29.146656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:29.147385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:29.148809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6dc87af-48ca-449b-8eeb-ae21b0495d87 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:29.173019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 591/1173] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:49:29 2022] POST /identity/v3/projects => generated 361 bytes in 159 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:49:31.213866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93671f6-d4cc-45a5-b71f-eaf4df5338ba None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:31.214423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93671f6-d4cc-45a5-b71f-eaf4df5338ba None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:31.214423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f93671f6-d4cc-45a5-b71f-eaf4df5338ba None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:31.215823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 583/1174] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:31.220922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:31.221360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:31.221488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:31.239986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:31.241163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:31.305566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 592/1175] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:31 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:31.329960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93671f6-d4cc-45a5-b71f-eaf4df5338ba None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:31.341012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:31.429900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f93671f6-d4cc-45a5-b71f-eaf4df5338ba None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:31.462150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:31.463036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:31.463320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:31.463595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] PATH_INFO: `/v3/domains/53a0948fd9ca455692470d4878e97fa1` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:31.464812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] RBAC: Authorizing `identity:get_domain(domain_id=53a0948fd9ca455692470d4878e97fa1)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:31.466171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c29d7c7-9583-4ef9-87c3-2c580b830e53 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:31.467827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 584/1176] 10.222.0.18 () {56 vars in 1183 bytes} [Sun Jul 24 00:49:31 2022] GET /identity/v3/domains/53a0948fd9ca455692470d4878e97fa1 => generated 267 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:31.474651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:31.567603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:31.571572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ff0d585-1913-4887-8526-e0cf8223b057 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:31.606333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:31.607017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:31.607299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:31.607572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:31.608266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:31.609682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9646f4ff-bccd-42e5-b60b-a14c1a0ba9bd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:31.628341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 593/1177] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:49:31 2022] POST /identity/v3/users => generated 311 bytes in 154 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:49:33.696818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c28aa-ec98-4195-9a15-7299d5e5ec46 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:33.697375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c28aa-ec98-4195-9a15-7299d5e5ec46 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:33.697375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a34c28aa-ec98-4195-9a15-7299d5e5ec46 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:33.698677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 585/1178] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:33.703823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:33.704274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:33.704380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:33.720602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:33.721805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:33.786202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 594/1179] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:33 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:33.809352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c28aa-ec98-4195-9a15-7299d5e5ec46 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:33.888576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:33.916351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a34c28aa-ec98-4195-9a15-7299d5e5ec46 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:33.948453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:33.949174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:33.949459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:33.949737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:33.961128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:33.968777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:33.969957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:49:33.970967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 586/1180] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:33 2022] GET /identity/v3/roles/admin => generated 83 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:33.976786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:34.058079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:34.075058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35f39eba-e4bc-4203-aa22-13f873d0b3e8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:34.108848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:34.109499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:34.109779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:34.110049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:34.110797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:34.111106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:34.112460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:34.120564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 595/1181] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:33 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:34.126291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:34.223404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48934e24-e71c-4ecd-b38e-f43e2d0f1923 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:34.256310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:34.257173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:34.257454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:34.257728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] PATH_INFO: `/v3/users/fac169cd9cf74f228c7d57a18b7f83d8` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:34.260334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:34.271392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] RBAC: Authorizing `identity:get_user(user_id=fac169cd9cf74f228c7d57a18b7f83d8)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:34.272830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:34.274477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 587/1182] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:34 2022] GET /identity/v3/users/fac169cd9cf74f228c7d57a18b7f83d8 => generated 311 bytes in 149 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:34.280325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:34.377705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ff6b343-7195-496c-8422-79a989810d57 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:34.411331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:34.411975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:34.412254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:34.412527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:34.427611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:34.478990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:34.480365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:34.576127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 596/1183] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:34 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:34.582259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:34.677595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7922c16f-8949-484c-8632-fc0b8b7fb297 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:34.710228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:34.711105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:34.711387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:34.711661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] PATH_INFO: `/v3/projects/71bd65771dbf449985a4dcfa68222d9b` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:34.712826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] RBAC: Authorizing `identity:get_project(project_id=71bd65771dbf449985a4dcfa68222d9b)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:34.714243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77a062c4-6a2a-4cbf-a0c0-a825ae927dbe None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:34.715945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 588/1184] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:34 2022] GET /identity/v3/projects/71bd65771dbf449985a4dcfa68222d9b => generated 361 bytes in 134 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:34.721863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:34.766715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:34.818496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-304f704d-189a-49b1-ae64-767f16bfe94e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:34.850383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:34.851052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:34.851334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:34.851607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:34.852423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=71bd65771dbf449985a4dcfa68222d9b, user.id=fac169cd9cf74f228c7d57a18b7f83d8) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:34.852732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:34.854152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91fe32fc-c698-49f7-bdf9-75a785b5c5a9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:34.859430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 597/1185] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:34 2022] GET /identity/v3/role_assignments?user.id=fac169cd9cf74f228c7d57a18b7f83d8&scope.project.id=71bd65771dbf449985a4dcfa68222d9b&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:36.845613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bb1b3-5d1b-4292-82ef-13772df12385 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:36.846116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bb1b3-5d1b-4292-82ef-13772df12385 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:36.846116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-978bb1b3-5d1b-4292-82ef-13772df12385 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:36.848096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 589/1186] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:36.853286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:36.853701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:36.853828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:36.873025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:36.874189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:36.940580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 598/1187] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:36.964075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978bb1b3-5d1b-4292-82ef-13772df12385 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:37.024447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:37.063178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-978bb1b3-5d1b-4292-82ef-13772df12385 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:37.095686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:37.096412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:37.096698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:37.096975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:37.106937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:37.115274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:37.116404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:49:37.117473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 590/1188] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:36 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:37.122607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:37.163654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:37.218959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f80ef2-df56-4faf-a52a-8b31288a58cf None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:37.252168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:37.252826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:37.253107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:37.253383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:37.254148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:37.254450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:37.255847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:37.263612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 599/1189] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:37 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:37.269916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:37.366695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01af7b96-ea54-4d4f-85ab-bdb57ea998d6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:37.387447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:37.399338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:37.400201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:37.400478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:37.400750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] PATH_INFO: `/v3/users/fac169cd9cf74f228c7d57a18b7f83d8` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:37.401909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] RBAC: Authorizing `identity:get_user(user_id=fac169cd9cf74f228c7d57a18b7f83d8)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:37.403372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:37.404983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 591/1190] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:37 2022] GET /identity/v3/users/fac169cd9cf74f228c7d57a18b7f83d8 => generated 311 bytes in 136 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:37.411922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:37.509343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5799f4-1aac-4286-9d31-e951cc234f22 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:37.510840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:37.543900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:37.544749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:37.545031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:37.545305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] PATH_INFO: `/v3/projects/71bd65771dbf449985a4dcfa68222d9b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:37.546476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] RBAC: Authorizing `identity:get_project(project_id=71bd65771dbf449985a4dcfa68222d9b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:37.547939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7481f3e1-6bc3-419a-8410-731d6494631b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:37.549606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 600/1191] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:37 2022] GET /identity/v3/projects/71bd65771dbf449985a4dcfa68222d9b => generated 361 bytes in 138 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:49:37.555517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:37.650939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3cdd7e-9cd8-4a02-b768-cd6b1a9be865 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:37.682752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:37.683650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:37.683933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:37.684206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] PATH_INFO: `/v3/projects/71bd65771dbf449985a4dcfa68222d9b/users/fac169cd9cf74f228c7d57a18b7f83d8/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:37.686378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] RBAC: Authorizing `identity:create_grant(project_id=71bd65771dbf449985a4dcfa68222d9b, user_id=fac169cd9cf74f228c7d57a18b7f83d8, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:37.687826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6f29a69-066f-4965-bc92-dada743be655 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:37.699942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 592/1192] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:49:37 2022] PUT /identity/v3/projects/71bd65771dbf449985a4dcfa68222d9b/users/fac169cd9cf74f228c7d57a18b7f83d8/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 145 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:49:39.696030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193153e3-56e6-4e68-951a-7199f11ccebf None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:39.696555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193153e3-56e6-4e68-951a-7199f11ccebf None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:39.696555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-193153e3-56e6-4e68-951a-7199f11ccebf None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:39.697886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 601/1193] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:49:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:49:39.702953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:39.703361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:39.703511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:39.720700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:49:39.721875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:49:39.766452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:39.788736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 593/1194] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:49:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:49:39.811897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193153e3-56e6-4e68-951a-7199f11ccebf None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:39.911245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-193153e3-56e6-4e68-951a-7199f11ccebf None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:39.943654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:39.944436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:39.944721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:39.945001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:39.954875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:49:39.963098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:49:39.964208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:49:39.965252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 602/1195] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:49:39 2022] GET /identity/v3/roles/admin => generated 83 bytes in 153 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:49:39.970201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:40.065529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc79c4e-44f1-4261-b2fd-24e7fa7854c3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:40.099240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:40.099899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:40.100183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:40.100458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:40.101195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:40.101505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:40.102920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:40.110812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 594/1196] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:49:39 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:40.116690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:40.148876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:40.216952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32b787e0-d062-469b-9c36-1bb714ebabba None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:40.249473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:40.250334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:40.250617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:40.250923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] PATH_INFO: `/v3/users/fac169cd9cf74f228c7d57a18b7f83d8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:40.252088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] RBAC: Authorizing `identity:get_user(user_id=fac169cd9cf74f228c7d57a18b7f83d8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:40.254053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:40.255785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 603/1197] 10.222.0.18 () {56 vars in 1177 bytes} [Sun Jul 24 00:49:40 2022] GET /identity/v3/users/fac169cd9cf74f228c7d57a18b7f83d8 => generated 311 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:40.261477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:40.303439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:40.358422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-182d0180-0743-4842-ad07-dbf9ea18fa6c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:40.390218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:40.390888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:40.391184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:40.391460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:40.457146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:40.458732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:40.497588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:40.552259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 595/1198] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:49:40 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:49:40.558122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:40.651763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:40.655272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b32dd4a-f7d3-4572-be78-cae188de0210 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:40.688010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:40.688852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:40.689132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:40.689404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] PATH_INFO: `/v3/projects/71bd65771dbf449985a4dcfa68222d9b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:40.690552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] RBAC: Authorizing `identity:get_project(project_id=71bd65771dbf449985a4dcfa68222d9b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:40.692606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f82eb8c-0f27-437e-aa4b-982fd24af09b None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:40.694257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 604/1199] 10.222.0.18 () {56 vars in 1186 bytes} [Sun Jul 24 00:49:40 2022] GET /identity/v3/projects/71bd65771dbf449985a4dcfa68222d9b => generated 361 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:40.700410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:49:40.800077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2962c44-2c47-487f-b81c-952fcbabd019 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:49:40.833336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:49:40.833972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:49:40.834264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:49:40.834543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:49:40.835388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=71bd65771dbf449985a4dcfa68222d9b, user.id=fac169cd9cf74f228c7d57a18b7f83d8) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:49:40.835703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:49:40.837121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94952640-e0f6-4a53-a167-26e96f70e804 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:49:40.842713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 596/1200] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:49:40 2022] GET /identity/v3/role_assignments?user.id=fac169cd9cf74f228c7d57a18b7f83d8&scope.project.id=71bd65771dbf449985a4dcfa68222d9b&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:49:50.683845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=17 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:49:50.840181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:36.082301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa461188-e299-484f-acdd-d819e8f129cc None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:36.082301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa461188-e299-484f-acdd-d819e8f129cc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:36.082301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa461188-e299-484f-acdd-d819e8f129cc None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:36.082301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 605/1201] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:36 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:36.091392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:36.091881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:36.091971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:36.113674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:36.115367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:36.210390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 597/1202] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:36 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 120 msecs (HTTP/1.1 201) 6 headers in 364 bytes (2 switches on core 0) Jul 24 00:50:36.236374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa461188-e299-484f-acdd-d819e8f129cc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:36.369876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa461188-e299-484f-acdd-d819e8f129cc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:36.420037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:36.420858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:36.421177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:36.421451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:36.434266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:36.437586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:36.461183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:36.462497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:36.463907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:36.463907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:36.463907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:36.463907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:36.463907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 606/1203] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:36 2022] GET /identity/v3/domains/Default => generated 87 bytes in 228 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:36.468997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:36.608060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e84056f-9af2-446d-b4a6-e212e96c5a0f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:36.649077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:36.662281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:36.663127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:36.663424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:36.663734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:36.664498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:36.664875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:36.666411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2f280ca-05fb-4cb5-ab5f-6aacc8a7c6c4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:36.682045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 598/1204] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 214 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:36.688609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:36.760363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:36.815491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22c04e52-6cc2-4f49-8caa-873081dcb9e3 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:36.871697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:36.872473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:36.872898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:36.873193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:36.873972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:36.875484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fd77c8f-232c-4cc3-9665-12c7fcd95874 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:36.905519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 607/1205] 10.222.0.18 () {60 vars in 1132 bytes} [Sun Jul 24 00:50:36 2022] POST /identity/v3/users => generated 251 bytes in 218 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:50:39.283797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db70750-053c-4b87-8007-501ca31442f3 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:39.284368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db70750-053c-4b87-8007-501ca31442f3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:39.284368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3db70750-053c-4b87-8007-501ca31442f3 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:39.285719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 599/1206] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:39 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:39.291380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:39.291874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:39.291980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:39.311497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:39.312690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:39.394668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 608/1207] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:39 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:50:39.420399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db70750-053c-4b87-8007-501ca31442f3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:39.538133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3db70750-053c-4b87-8007-501ca31442f3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:39.575705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:39.576505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:39.576817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:39.577081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:39.588519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:50:39.596784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:39.597788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:39.598665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 600/1208] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:50:39 2022] GET /identity/v3/roles/service => generated 85 bytes in 178 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:39.604276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:39.652577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:39.722999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-092d89e5-3d17-4633-bce1-eceaafede675 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:39.762371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:39.763084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:39.763369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:39.763643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:39.764485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:39.764926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:39.767252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:39.781667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 609/1209] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:50:39 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:39.784042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:39.838398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:39.901536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7eba8c6c-e76f-4b39-8c86-6988e20d62c8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:39.944171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:39.945795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:39.945795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:39.945795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:39.958976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:39.960517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:39.980369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:39.983325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:39.984229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:39.984229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:39.984229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:39.984229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:39.984229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 601/1210] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:39 2022] GET /identity/v3/domains/Default => generated 87 bytes in 198 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:39.989966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:40.101937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:40.104927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-848e5893-e618-4189-bb51-82552adfe56f None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:40.144911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:40.145860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:40.146280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:40.146410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:40.147194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:40.147507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:40.148871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:40.162287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 610/1211] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 173 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:40.168162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:40.287883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b08538-5192-4f50-9fc4-cc35a8e2f405 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:40.289465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:40.328942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:40.329755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:40.330028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:40.330298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:40.342839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:40.343810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:50:40.361663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:40.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:40.364153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 602/1212] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:40 2022] GET /identity/v3/users/placement => generated 87 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:50:40.369143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:40.489473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4079fca-af41-43fb-b1f4-bfaec9b59837 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:40.530337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:40.531191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:40.531539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:40.531932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:40.533297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:40.533297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:40.534976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:40.556995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 611/1213] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:50:40 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 188 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:40.564413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:40.680933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72c0e727-3280-4b17-a297-951336d7ead5 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:40.717509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:40.718161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:40.718483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:40.718790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:40.756471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:40.803746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:40.805203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:40.930663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 603/1214] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:50:40 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 368 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:50:40.937328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:40.960569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:41.057297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9092c492-63ea-4836-808c-9a62c74573bd None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:41.096989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:41.097793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:41.098079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:41.098355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:41.108957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:41.110992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:41.132575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:41.133825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:41.135019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:41.135019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:41.135019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:41.135019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:41.136645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 612/1215] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:40 2022] GET /identity/v3/domains/Default => generated 87 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:41.142954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:41.216566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:41.267896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd8dd465-a71a-4545-8971-277af02ccc51 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:41.307113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:41.307808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:41.308131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:41.308354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:41.309168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:41.309834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:41.311233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:41.331041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 604/1216] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 184 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:41.332816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:41.414389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:41.452487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b6db35e-5c8d-42ec-9f07-eca680571651 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:41.502420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:41.503403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:41.503690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:41.503961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:41.513012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:41.514866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:50:41.530807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:41.531892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:41.533051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 613/1217] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:50:41 2022] GET /identity/v3/projects/service => generated 88 bytes in 200 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:41.538909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:41.663216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05e0f687-3a7d-4a4e-b7c8-67486e67c818 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:41.689423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:41.704083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:41.704967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:41.705306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:41.705704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:41.706502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:41.707030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:41.708763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d7eef1-d4c6-4872-9af6-75a70d211f52 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:41.729001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 605/1218] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:50:41 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 186 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:41.730147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:41.852740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf4b5c6-217e-4dd6-af7f-cff4c213968c None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:41.888003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:41.895396 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:41.896102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:41.896434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:41.896711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:41.897551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=9eaff003925c4f7283d99ad51db217e2) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:41.897850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:41.899574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d36f1094-bcc4-4193-9084-895bacb1bd20 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:41.914804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 614/1219] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:50:41 2022] GET /identity/v3/role_assignments?user.id=9eaff003925c4f7283d99ad51db217e2&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 259 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:43.995983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae4140-c338-4730-b0f7-4e3f376d7d65 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:43.996547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae4140-c338-4730-b0f7-4e3f376d7d65 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:43.996547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1eae4140-c338-4730-b0f7-4e3f376d7d65 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:43.998279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 606/1220] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:43 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:44.004350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.005733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.005733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.022928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:44.024109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:44.094343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 615/1221] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:44 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:50:44.117499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae4140-c338-4730-b0f7-4e3f376d7d65 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:44.226271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1eae4140-c338-4730-b0f7-4e3f376d7d65 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:44.261883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:44.262680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.263058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.263344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] PATH_INFO: `/v3/roles/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.274784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:50:44.283794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:44.285185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:44.286304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:44.290962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 607/1222] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:50:44 2022] GET /identity/v3/roles/service => generated 85 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:44.292002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:44.400594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b1bf804-05a3-4222-95a7-f93f36fe41b7 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:44.427099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:44.439269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:44.439945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.440215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.440485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.441225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] RBAC: Adding query filter params (name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:44.441525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:44.442918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:44.452130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 616/1223] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:50:44 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 161 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:44.458450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:44.506945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:44.569666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bed3aacb-24df-4265-9ccb-fab557219a4e None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:44.604485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:44.605275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.605556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.605829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.618307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:44.619801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:44.637298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:44.638391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:44.639527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:44.639527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:44.639527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:44.639527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:44.639527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 608/1224] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:44 2022] GET /identity/v3/domains/Default => generated 87 bytes in 181 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:44.646007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:44.758643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf58cee-69dd-4abf-ab64-cfe7feebbc16 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:44.794177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:44.795123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.796668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.796668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.796937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:44.798043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:44.799826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:44.813509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 617/1225] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:44 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:44.819232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:44.928534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:44.937306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbc6358f-5830-459f-9d86-2fed042f995a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:44.977925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:44.978923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:44.979234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:44.979564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:44.990320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:44.992335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:50:45.009843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:45.011564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:45.012850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 609/1226] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:44 2022] GET /identity/v3/users/placement => generated 87 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:45.021791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:45.075397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:45.132500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0899549-303a-4689-b9a0-618fe11787ef None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:45.173681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:45.174367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:45.174633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:45.175066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:45.175975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:45.176516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:45.177824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.206509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 618/1227] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:50:45 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 185 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:45.212191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:45.323281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1210682-a791-4a36-b7d7-f725695a5f55 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:45.359925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:45.360774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:45.361149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:45.361598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:45.372577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:45.374056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:45.392833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:45.394505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:45.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:45.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:45.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:45.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:45.395674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 610/1228] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:45 2022] GET /identity/v3/domains/Default => generated 87 bytes in 182 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:45.400081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:45.518527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a470931-b879-4c6a-9345-956335f0eaac None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:45.521117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:45.560314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:45.561066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:45.561351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:45.561715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:45.562442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:45.562853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:45.564232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.576680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 619/1229] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:45 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:45.584141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:45.594320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:45.696821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6701d0-458f-4c82-a45a-1593885e9a9a None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:45.732502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:45.733249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:45.733531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:45.733810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:45.740907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:45.742357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:50:45.756751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:45.757891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:45.758933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 611/1230] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:50:45 2022] GET /identity/v3/projects/service => generated 88 bytes in 174 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:45.764403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:45.879957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ddaf7ea-9ea8-4d90-94c1-9c4d129e42a0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:45.914581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:45.915363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:45.915685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:45.915989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:45.916777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:45.917131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:45.918735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49e49e87-e4a3-45db-b191-529b834e588a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:45.933037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 620/1231] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:50:45 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 169 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:45.939894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:46.011515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:46.047079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6a6ecb7-9103-4df1-9171-e37a8bfe1939 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:46.082586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:46.083553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:46.083802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:46.084077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/9eaff003925c4f7283d99ad51db217e2/roles/e06beecc102046d8861a0db046c60528` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:46.099965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=9eaff003925c4f7283d99ad51db217e2, role_id=e06beecc102046d8861a0db046c60528)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:46.101417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccee1274-6166-4fff-beaf-72ef9e2fde9a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:46.114941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 612/1232] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:50:45 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/9eaff003925c4f7283d99ad51db217e2/roles/e06beecc102046d8861a0db046c60528 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:50:48.459021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73a76-187f-4156-9350-aed1aea580de None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:48.459595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73a76-187f-4156-9350-aed1aea580de None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:48.459595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5b73a76-187f-4156-9350-aed1aea580de None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:48.460923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 621/1233] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:48 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:48.467022 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:48.467490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:48.467622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:48.486321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:48.487586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:48.560277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 613/1234] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:48 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:50:48.587479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b73a76-187f-4156-9350-aed1aea580de None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:48.617129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:48.700564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5b73a76-187f-4156-9350-aed1aea580de None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:48.736901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810899be-08be-4996-bf92-325e18f561ce None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:48.737795 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810899be-08be-4996-bf92-325e18f561ce None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:48.738076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810899be-08be-4996-bf92-325e18f561ce None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:48.738345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-810899be-08be-4996-bf92-325e18f561ce None admin] PATH_INFO: `/v3/roles/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:48.749753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-810899be-08be-4996-bf92-325e18f561ce None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-810899be-08be-4996-bf92-325e18f561ce None admin] Could not find role: service.: keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:50:48.758277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:48.759516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: service. Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:48.760621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 622/1235] 10.222.0.18 () {56 vars in 1102 bytes} [Sun Jul 24 00:50:48 2022] GET /identity/v3/roles/service => generated 85 bytes in 173 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:48.767002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:48.816906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:48.877446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6900f996-27ba-4e18-aa93-ce6a7ee7fa84 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:48.915167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:48.915839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:48.916118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:48.916393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:48.917137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] RBAC: Adding query filter params (name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:48.917447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:48.918835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:48.928954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 614/1236] 10.222.0.18 () {56 vars in 1103 bytes} [Sun Jul 24 00:50:48 2022] GET /identity/v3/roles?name=service => generated 326 bytes in 163 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:48.935479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810899be-08be-4996-bf92-325e18f561ce None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:49.028231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:49.047482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-810899be-08be-4996-bf92-325e18f561ce None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:49.089077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:49.090121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:49.090383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:49.090676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:49.102610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:49.104208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:49.123270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:49.124739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:49.126195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:49.126195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:49.126195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:49.126195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:49.126195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 623/1237] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:48 2022] GET /identity/v3/domains/Default => generated 87 bytes in 190 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:50:49.131595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:49.248047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-531553a9-96c4-4e58-b8bf-d63da23221b8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:49.259087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:49.287613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:49.288399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:49.288730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:49.289038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:49.289846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:49.290200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:49.291839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:49.305059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 615/1238] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:49.311548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:49.424972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4008d29a-0d16-4fe5-bdbe-ef3cad075904 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:49.464761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:49.465994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:49.465994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:49.466264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:49.469970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:49.477126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:49.479383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:50:49.496673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:49.497723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:49.498659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 624/1239] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:49 2022] GET /identity/v3/users/placement => generated 87 bytes in 187 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:49.504455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:49.613038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45a4ef2b-d7a5-424c-8bd4-38f941e4eae2 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:49.648331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:49.648996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:49.649288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:49.649559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:49.650388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:49.650699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:49.652359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:49.676679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 616/1240] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:50:49 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:49.678217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:49.792925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757958-5a80-45c0-8404-6f94667b01a8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:49.828705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:49.829609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:49.829878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:49.830161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:49.898347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:49.905412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:49.907284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.027409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 625/1241] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:50:49 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 350 msecs (HTTP/1.1 200) 6 headers in 359 bytes (2 switches on core 0) Jul 24 00:50:50.033478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:50.061519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:50.137771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19da052c-71d2-4a13-b9d5-9f0d42ec0f6f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:50.174635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:50.175638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:50.176013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:50.176375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:50.188568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:50.190360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:50.211338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:50.212490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:50.213552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:50.213552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:50.213552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:50.213552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:50.213552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 617/1242] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:50 2022] GET /identity/v3/domains/Default => generated 87 bytes in 180 msecs (HTTP/1.1 404) 5 headers in 183 bytes (2 switches on core 0) Jul 24 00:50:50.218617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:50.327518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:50.334272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c96c9dd-a635-4473-b2c5-d635983241cb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:50.374735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:50.376045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:50.376322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:50.377341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:50.377341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:50.377637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:50.379100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.392485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 626/1243] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 174 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:50.398542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:50.486163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:50.510706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9104573-4bc2-4750-b617-7cf2ab097212 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:50.547878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:50.548596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:50.548877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:50.549154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:50.555970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:50.557446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:50:50.573184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:50.574474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:50.575629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 618/1244] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:50:50 2022] GET /identity/v3/projects/service => generated 88 bytes in 177 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:50.581373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:50.693158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b343dede-0bb5-43e7-a471-bacb59780ec8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:50.735805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:50.736598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:50.736828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:50.737106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:50.737895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:50.738208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:50.739682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1ef189-cd2b-4a76-9677-36e6205b65fd None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.755990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 627/1245] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:50:50 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 175 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:50.762001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:50.873774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-208b92ca-c73e-4a1d-b90c-91295776ed3f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:50.910403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:50.911106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:50.911393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:50.911679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:50.912513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] RBAC: Adding query filter params (role.id=e06beecc102046d8861a0db046c60528, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=9eaff003925c4f7283d99ad51db217e2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:50.912845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:50.914262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5648c32a-fff5-475f-a3c1-62b4dedbc50d None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:50.920540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 619/1246] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:50:50 2022] GET /identity/v3/role_assignments?user.id=9eaff003925c4f7283d99ad51db217e2&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=e06beecc102046d8861a0db046c60528 => generated 608 bytes in 159 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:53.252234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989df2d-c1e3-48c0-b8d0-b29532506fe8 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:53.252799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989df2d-c1e3-48c0-b8d0-b29532506fe8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:53.252799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989df2d-c1e3-48c0-b8d0-b29532506fe8 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:53.254307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 628/1247] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:53.260336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:53.260774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:53.261029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:53.267499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:53.280275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:53.282692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:53.361830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 620/1248] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:53 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:50:53.387046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989df2d-c1e3-48c0-b8d0-b29532506fe8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:53.407526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:53.503069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989df2d-c1e3-48c0-b8d0-b29532506fe8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:53.549441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:53.550233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:53.550543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:53.550922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:53.563116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:50:53.572410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:53.573548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:50:53.574634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:53.579899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 629/1249] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:50:53 2022] GET /identity/v3/roles/admin => generated 83 bytes in 193 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:53.581134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:53.698447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d21b12c5-22c1-4861-8c03-487d917ac5d2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:53.731503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:53.737558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:53.738317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:53.738671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:53.739027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:53.739786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:53.740095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:53.741460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:53.751613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 621/1250] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:50:53 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:53.757928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:53.841374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:53.866915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca5af6ff-e596-4c28-b0f6-ce826848d67e None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:53.904874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:53.905669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:53.905982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:53.906247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:53.918502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:53.920013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:53.939283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:53.940650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:53.941739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:53.941739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:53.941739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:53.941739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:53.941739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 630/1251] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:53 2022] GET /identity/v3/domains/Default => generated 87 bytes in 183 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:53.946813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:54.066304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b385088-ed2c-4dbe-8166-ab5010d9ee8c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:54.104643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:54.105445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:54.107135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:54.107135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:54.107135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:54.107378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:54.108895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:54.122268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 622/1252] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:53 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 176 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:54.129859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:54.251859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab403e8-d6bd-4a47-867e-d3d1b15f7f12 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:54.293034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:54.301897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:54.302722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:54.303038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:54.303315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:54.313031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:54.314876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:50:54.335491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:54.336676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:54.338230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:54.342806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 631/1253] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:54 2022] GET /identity/v3/users/placement => generated 87 bytes in 213 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:54.343243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:54.416122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:54.467621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8157fa5c-eabe-40c6-80d4-1dbc2f80b3f3 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:54.510034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:54.510704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:54.511018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:54.511295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:54.512147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:54.512458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:54.513899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:54.534514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 623/1254] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:50:54 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 192 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:50:54.540333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:54.656452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c73378e3-30f2-4221-8fc0-5643320d0fc6 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:54.692166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:54.692835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:54.693154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:54.693433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:54.763435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:54.781615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:54.783649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:54.903611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 632/1255] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:50:54 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 364 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:50:54.910638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:55.036961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2e86d8c-b2bc-4178-80bf-d503d33811bb None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:55.053973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:55.079733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:55.080794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:55.081084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:55.081367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:55.092772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:55.094994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:55.116216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:55.118184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:55.124060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:55.124060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:55.124060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:55.124060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:55.124060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 624/1256] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:54 2022] GET /identity/v3/domains/Default => generated 87 bytes in 208 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:55.124344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:55.227386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:55.240621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3f9950f-edf5-49a1-920e-23bb29816075 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:55.279027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:55.279694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:55.279974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:55.280261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:55.281062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:55.281423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:55.282838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:55.294344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 633/1257] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 171 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:55.300426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:55.429348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8adae0ab-ceac-4079-8356-5d5c3f15ed8f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:55.463790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:55.464595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:55.464880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:55.465156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:55.473401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:55.474875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:50:55.493197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:55.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:50:55.495177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 625/1258] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:50:55 2022] GET /identity/v3/projects/service => generated 88 bytes in 195 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:55.501467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:55.615181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-077e06b4-0f8e-489c-b898-0e1af0778de8 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:55.650304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:55.651025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:55.651312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:55.651589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:55.652402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:55.652715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:55.654197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f18fac8-cc03-437f-954f-c857c46409f4 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:55.668197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 634/1259] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:50:55 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 168 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:55.674603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:55.702402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:55.781890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c3733f-96d3-4a5d-bfaa-034e1259fe04 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:55.817244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:55.817905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:55.818193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:55.818471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:55.819355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=9eaff003925c4f7283d99ad51db217e2) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:55.819672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:55.821155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7b18077-c0ca-4b7a-9538-f291a0480419 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:55.827323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 626/1260] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:50:55 2022] GET /identity/v3/role_assignments?user.id=9eaff003925c4f7283d99ad51db217e2&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 259 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:57.829725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be131f-c411-4c35-8ff1-9f5405f23aa1 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:57.830248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be131f-c411-4c35-8ff1-9f5405f23aa1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:57.830248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53be131f-c411-4c35-8ff1-9f5405f23aa1 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:57.831638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 635/1261] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:50:57 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:50:57.836744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:57.837109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:57.837302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:57.854822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:50:57.855994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:50:57.919427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 627/1262] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:50:57 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:50:57.942365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be131f-c411-4c35-8ff1-9f5405f23aa1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:57.953660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.043010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53be131f-c411-4c35-8ff1-9f5405f23aa1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.075751 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.076542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.076827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.077105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.087907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:50:58.096273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:58.097367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:58.098601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 636/1263] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:50:57 2022] GET /identity/v3/roles/admin => generated 83 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:58.103518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.121207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.212046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6376f63-8002-4e3f-b7af-74973690c2b1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.248427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.249086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.249858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.250130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.250905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:58.251211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:58.252588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:58.260360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 628/1264] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:58.266655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.330914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.363987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc53958-3f99-47fb-91bd-f4fe8422459a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.395277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.396063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.396341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.396611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.407622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:58.409052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:58.426385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:58.427471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:58.428581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:58.428581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:58.428581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:58.428581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:58.428581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 637/1265] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 161 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:58.433329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.511004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.530194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7fd2d605-d999-4955-a9aa-f929396b50e6 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.562452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.563114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.563398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.563671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.564380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:58.564685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:58.566021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:58.577227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 629/1266] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 144 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:58.583398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.679843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b9f823a-2495-4817-a101-0e7e80fb9d02 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.699766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.713854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.714639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.714954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.715230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] PATH_INFO: `/v3/users/placement` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.723138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:58.724566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:50:58.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:58.739681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:58.741017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 638/1267] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/users/placement => generated 87 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:58.745455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.840197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bbc71fa-3a2f-4839-b604-cf68716d8e53 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:58.872673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:58.873325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:58.873608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:58.873887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:58.875626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:58.875944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:58.877362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:58.885593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:58.895069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 630/1268] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:58.900673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:58.996195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ddd9af2-3ce7-4092-8ebf-af5ecfc1caec None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:59.029308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:59.030088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:59.030367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:59.030638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:59.041108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:59.042536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:50:59.059580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:59.060640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:59.062561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:50:59.062561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:50:59.062561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:50:59.062561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:59.062561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 639/1269] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:50:58 2022] GET /identity/v3/domains/Default => generated 87 bytes in 160 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:59.067072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:59.166106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd84bf81-500e-451e-b6c7-a6160431c542 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:59.202199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:59.203571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:59.203819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:59.204094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:59.204801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:59.205104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:59.206458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:59.217213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 631/1270] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:50:59 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:59.223521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:59.254819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:59.321940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4e3463a6-a1aa-45db-abbe-7433eb30e333 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:59.355304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:59.356087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:59.356371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:59.356646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:59.363219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:59.364658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:50:59.378714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:50:59.379797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:50:59.380942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 640/1271] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:50:59 2022] GET /identity/v3/projects/service => generated 88 bytes in 157 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:50:59.385808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:59.444061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:59.484441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f2288f-edc2-43a1-8f3a-b9d6b9549349 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:59.516373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:59.516958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:59.517237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:59.517510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:59.518277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:50:59.518584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:59.520025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-450cf8a2-7aa1-40e6-8ed8-17853d45dee7 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:59.531984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 632/1272] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:50:59 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:50:59.540082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:50:59.631351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:50:59.640807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19743265-02d7-4025-ac80-74abb2cf9862 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:50:59.673025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:50:59.673903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:50:59.674181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:50:59.674455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] PATH_INFO: `/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/9eaff003925c4f7283d99ad51db217e2/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:50:59.676557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] RBAC: Authorizing `identity:create_grant(project_id=ca6929ff61c44a8bb05ab68c6be32354, user_id=9eaff003925c4f7283d99ad51db217e2, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:50:59.677991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cde6a546-d050-47df-958a-ae28aaf68d3f None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:50:59.691242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 641/1273] 10.222.0.18 () {58 vars in 1439 bytes} [Sun Jul 24 00:50:59 2022] PUT /identity/v3/projects/ca6929ff61c44a8bb05ab68c6be32354/users/9eaff003925c4f7283d99ad51db217e2/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:51:01.676014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80791a1f-1940-4ff7-87eb-fd6eda2a1bb6 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:01.676516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80791a1f-1940-4ff7-87eb-fd6eda2a1bb6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:01.676516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80791a1f-1940-4ff7-87eb-fd6eda2a1bb6 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:01.677867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 633/1274] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:51:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:51:01.683078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:01.683502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:01.683634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:01.700117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:51:01.701289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:51:01.765409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 642/1275] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:51:01 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 83 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:51:01.788365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80791a1f-1940-4ff7-87eb-fd6eda2a1bb6 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:01.885625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80791a1f-1940-4ff7-87eb-fd6eda2a1bb6 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:01.902015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:01.918793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:01.919588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:01.919869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:01.920145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] PATH_INFO: `/v3/roles/admin` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:01.930595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] Extracting inferred target data resulted in "NOT FOUND (404)". {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:409}} Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] Could not find role: admin.: keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 50, in get Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(role_id) Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/roles.py", line 70, in _get_role Jul 24 00:51:01.938917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ENFORCER.enforce_call(action='identity:get_role') Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 402, in enforce_call Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application policy_dict.update(cls._extract_member_target_data( Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py", line 234, in _extract_member_target_data Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application member_name: func(flask.request.view_args[key]) Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:01.940037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/core.py", line 1276, in get_role Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_role(role_id) Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 70, in get_role Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_role(session, role_id).to_dict() Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/assignment/role_backends/sql.py", line 65, in _get_role Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.RoleNotFound(role_id=role_id) Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.RoleNotFound: Could not find role: admin. Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:01.941078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 634/1276] 10.222.0.18 () {56 vars in 1096 bytes} [Sun Jul 24 00:51:01 2022] GET /identity/v3/roles/admin => generated 83 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:01.946015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.024993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:02.043972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68d90476-d8ec-4ccb-9a64-aafe30944733 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.077245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.077998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.078282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.078561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.079334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] RBAC: Adding query filter params (name=admin) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:02.079645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.081014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:02.088856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 643/1277] 10.222.0.18 () {56 vars in 1099 bytes} [Sun Jul 24 00:51:01 2022] GET /identity/v3/roles?name=admin => generated 339 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:02.095158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.201241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e64f06d-6ad3-441b-a3d7-ddffff15bced None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.233559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.234952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.235509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.235786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.246071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.247532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:51:02.264038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:02.265688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:02.266614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:51:02.266614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:51:02.266614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:51:02.266614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:02.266614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 635/1278] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:51:02 2022] GET /identity/v3/domains/Default => generated 87 bytes in 171 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:02.271641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.369282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f583415d-c82d-46fc-8bb2-8cab35dba043 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.401659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.402317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.402600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.402906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.403629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:02.403933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.405298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:02.417547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 644/1279] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:51:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:02.423280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.453681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:02.520577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f86b1ab-5598-48ef-81a6-448776943e7f None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.552621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.553410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.553692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.553963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] PATH_INFO: `/v3/users/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.561362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] RBAC: Authorizing `identity:get_user(user_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.562896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] Could not find user: placement.: keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 182, in get Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_user(user_id) Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/users.py", line 194, in _get_user Jul 24 00:51:02.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = PROVIDERS.identity_api.get_user(user_id) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 414, in wrapper Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 424, in wrapper Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return f(self, *args, **kwargs) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:02.578390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/core.py", line 1011, in get_user Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application ref = driver.get_user(entity_id) Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 200, in get_user Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application self._get_user(session, user_id).to_dict()) Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/identity/backends/sql.py", line 194, in _get_user Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.UserNotFound(user_id=user_id) Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.UserNotFound: Could not find user: placement. Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:02.584149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 636/1280] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:51:02 2022] GET /identity/v3/users/placement => generated 87 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:02.584913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.588067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:02.681901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc65d1c0-6f79-4d01-a452-0dbe41399601 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.716170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.716764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.717047 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.717323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.718182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] RBAC: Adding query filter params (domain_id=default, name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:02.718486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] RBAC: Authorizing `identity:list_users()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.719949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:02.736399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 645/1281] 10.222.0.18 () {56 vars in 1143 bytes} [Sun Jul 24 00:51:02 2022] GET /identity/v3/users?domain_id=default&name=placement => generated 378 bytes in 153 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:02.742207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:02.831892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:02.841477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ad50d5-03fb-4fdf-a98c-76ce8dc23ee8 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:02.873353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04958260-2cd2-4c11-833c-4784446143af None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:02.873990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04958260-2cd2-4c11-833c-4784446143af None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:02.874822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04958260-2cd2-4c11-833c-4784446143af None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:02.875180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04958260-2cd2-4c11-833c-4784446143af None admin] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:02.939763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04958260-2cd2-4c11-833c-4784446143af None admin] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:02.941210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04958260-2cd2-4c11-833c-4784446143af None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.026515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:03.036479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 637/1282] 10.222.0.18 () {58 vars in 1282 bytes} [Sun Jul 24 00:51:02 2022] GET /identity/v3/auth/tokens => generated 2108 bytes in 295 msecs (HTTP/1.1 200) 6 headers in 359 bytes (1 switches on core 0) Jul 24 00:51:03.042464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:03.101146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:03.138078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d41ec7a4-4726-4554-8158-f08afc299bb0 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:03.174910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:03.175702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:03.176021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:03.176294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] PATH_INFO: `/v3/domains/Default` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:03.187262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] RBAC: Authorizing `identity:get_domain(domain_id=Default)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:03.189072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] Could not find domain: Default.: keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 89, in get Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_domain(domain_id) Jul 24 00:51:03.206266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/domains.py", line 97, in _get_domain Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application domain = PROVIDERS.resource_api.get_domain(domain_id) Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:03.207356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:03.208462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 716, in get_domain Jul 24 00:51:03.208462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.DomainNotFound(domain_id=domain_id) Jul 24 00:51:03.208462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.DomainNotFound: Could not find domain: Default. Jul 24 00:51:03.208462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:51:03.208462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 646/1283] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:51:03 2022] GET /identity/v3/domains/Default => generated 87 bytes in 165 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:03.213300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04958260-2cd2-4c11-833c-4784446143af None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:03.307695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04958260-2cd2-4c11-833c-4784446143af None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:03.340373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:03.341015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:03.341297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:03.341571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:03.342274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:03.342578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:03.344210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.359701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 638/1284] 10.222.0.18 () {56 vars in 1109 bytes} [Sun Jul 24 00:51:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 147 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:03.365233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:03.461460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3e530c4-5ba1-443c-b68d-968a152daaa1 None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:03.476504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:03.496550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:03.497331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:03.497613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:03.497886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] PATH_INFO: `/v3/projects/service` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:03.504514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] RBAC: Authorizing `identity:get_project(project_id=service)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:03.505951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] Could not find project: service.: keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 151, in get Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(project_id) Jul 24 00:51:03.519435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/projects.py", line 107, in _get_project Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application project = PROVIDERS.resource_api.get_project(project_id) Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:03.520468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/core.py", line 903, in get_project Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self.driver.get_project(project_id) Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 49, in get_project Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application return self._get_project(session, project_id).to_dict() Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/resource/backends/sql.py", line 44, in _get_project Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application raise exception.ProjectNotFound(project_id=project_id) Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application keystone.exception.ProjectNotFound: Could not find project: service. Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: ERROR keystone.server.flask.application Jul 24 00:51:03.521471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 647/1285] 10.222.0.18 () {56 vars in 1111 bytes} [Sun Jul 24 00:51:03 2022] GET /identity/v3/projects/service => generated 88 bytes in 156 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:03.526450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:03.580599 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:03.629253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e05751e-3542-4187-8a95-381719297f14 None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:03.661553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:03.662203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:03.662485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:03.662762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:03.663562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] RBAC: Adding query filter params (domain_id=default, name=service) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:03.663870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:03.665270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c885eb-15d1-401a-96a8-608f489d74e4 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.677786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 639/1286] 10.222.0.18 () {56 vars in 1148 bytes} [Sun Jul 24 00:51:03 2022] GET /identity/v3/projects?domain_id=default&name=service => generated 429 bytes in 152 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:03.684195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:03.783003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d98e4488-813a-4ffd-b7c1-0181b763c9fb None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:03.814992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:03.815626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:03.815904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:03.816177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] PATH_INFO: `/v3/role_assignments` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:03.816986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] RBAC: Adding query filter params (role.id=75e91b1e89304179904cf7644ef0a984, scope.project.id=ca6929ff61c44a8bb05ab68c6be32354, user.id=9eaff003925c4f7283d99ad51db217e2) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:03.817290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] RBAC: Authorizing `identity:list_role_assignments()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:03.818709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bcc6e52-69d5-473d-a34b-a064d4279f72 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:03.824322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 648/1287] 10.222.0.18 () {56 vars in 1374 bytes} [Sun Jul 24 00:51:03 2022] GET /identity/v3/role_assignments?user.id=9eaff003925c4f7283d99ad51db217e2&scope.project.id=ca6929ff61c44a8bb05ab68c6be32354&role.id=75e91b1e89304179904cf7644ef0a984 => generated 608 bytes in 141 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:05.881298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8203986-955e-4f21-82d3-59672a8eda65 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:05.881860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8203986-955e-4f21-82d3-59672a8eda65 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:05.881860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8203986-955e-4f21-82d3-59672a8eda65 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:05.883159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 640/1288] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:51:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:51:05.888181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:05.888577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:05.888736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:05.905781 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:51:05.906982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:51:05.935507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:05.972292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 649/1289] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:51:05 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:51:05.995000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8203986-955e-4f21-82d3-59672a8eda65 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:06.093221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8203986-955e-4f21-82d3-59672a8eda65 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:06.103400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:06.127497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:06.128363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:06.128647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:06.128922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] PATH_INFO: `/v3/services/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:06.129569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:06.130955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:06.143585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:06.144824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:06.145736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 641/1290] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:51:05 2022] GET /identity/v3/services/placement => generated 90 bytes in 150 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:06.151596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:06.266477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f1eeb24-c797-4ad4-9c56-be56640bd923 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:06.299341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:06.300001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:06.300286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:06.300564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:06.301263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:06.301574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:06.302969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11fa80f3-896b-48ef-920d-fa4eb93e4847 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:06.307719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 650/1291] 10.222.0.18 () {56 vars in 1116 bytes} [Sun Jul 24 00:51:06 2022] GET /identity/v3/services?name=placement => generated 126 bytes in 157 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:06.314403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:06.412383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbe92834-5ea6-448d-a2f0-2bcd0f95a4ca None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:06.445708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:06.446357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:06.446640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:06.446949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] PATH_INFO: `/v3/services` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:06.447647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] RBAC: Adding query filter params (type=placement) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:06.447955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:06.449316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18c711a4-333e-49e9-b94e-610f70527f71 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:06.453970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 642/1292] 10.222.0.18 () {56 vars in 1116 bytes} [Sun Jul 24 00:51:06 2022] GET /identity/v3/services?type=placement => generated 126 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:08.475580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f912a-dc51-4b24-82ff-37bb1845f1c2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:08.476086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f912a-dc51-4b24-82ff-37bb1845f1c2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:08.476184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f912a-dc51-4b24-82ff-37bb1845f1c2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:08.477498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 651/1293] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:51:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:51:08.482703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59882672-11e0-4fd1-b6f9-13848c69dade None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:08.483142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59882672-11e0-4fd1-b6f9-13848c69dade None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:08.483351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59882672-11e0-4fd1-b6f9-13848c69dade None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:08.499534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-59882672-11e0-4fd1-b6f9-13848c69dade None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:51:08.500999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-59882672-11e0-4fd1-b6f9-13848c69dade None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:51:08.529354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:08.566313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 643/1294] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:51:08 2022] POST /identity/v3/auth/tokens => generated 2108 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:51:08.589201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f912a-dc51-4b24-82ff-37bb1845f1c2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:08.619913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:08.696079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f912a-dc51-4b24-82ff-37bb1845f1c2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:08.729126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:08.729778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:08.730064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:08.730341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:08.731096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] RBAC: Authorizing `identity:create_service()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:08.732474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d647924f-4022-437b-89f3-e73e218cbb66 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:08.745036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 652/1295] 10.222.0.18 () {60 vars in 1141 bytes} [Sun Jul 24 00:51:08 2022] POST /identity/v3/services => generated 246 bytes in 156 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:51:10.789611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ede8a-a200-4837-9a61-aa9d78d15d1f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:10.790138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ede8a-a200-4837-9a61-aa9d78d15d1f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:10.790138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-072ede8a-a200-4837-9a61-aa9d78d15d1f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:10.791482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 644/1296] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:51:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:51:10.797156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:10.797554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:10.797719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:10.814892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:51:10.816082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:51:10.886885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 653/1297] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:51:10 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:51:10.909535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072ede8a-a200-4837-9a61-aa9d78d15d1f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:11.006613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-072ede8a-a200-4837-9a61-aa9d78d15d1f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:11.038935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:11.039755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:11.039984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:11.040255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] PATH_INFO: `/v3/services/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:11.040894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:11.042233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:11.054354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:11.055583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:11.056700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 645/1298] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:51:10 2022] GET /identity/v3/services/placement => generated 90 bytes in 146 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:11.061749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:11.079719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:11.160018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3846faf3-1708-429c-bbd9-bb3d5b9bddaf None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:11.201127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:11.201894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:11.202288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:11.202674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:11.203531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] RBAC: Adding query filter params (name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:11.203850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:11.205344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:11.211241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 654/1299] 10.222.0.18 () {56 vars in 1116 bytes} [Sun Jul 24 00:51:11 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 150 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:11.217266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:11.251458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:11.318196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6e7bf4-5775-4088-9da5-a86d454b2e4b None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:11.350739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:11.351415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:11.351696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:11.351967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:11.352762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] RBAC: Adding query filter params (interface=public, service_id=ded954a2fff5438fa054e317d7176bac, region_id=RegionOne) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:11.353070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] RBAC: Authorizing `identity:list_endpoints()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:11.354411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ca15940-794b-4f34-b91a-de7e498e1543 None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:11.359370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 646/1300] 10.222.0.18 () {56 vars in 1251 bytes} [Sun Jul 24 00:51:11 2022] GET /identity/v3/endpoints?service_id=ded954a2fff5438fa054e317d7176bac&interface=public®ion_id=RegionOne => generated 194 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:11.365053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:11.445964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:11.464548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7b7dd39-ce90-4153-b15e-f780daea339a None admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:11.497446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:11.498101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:11.498385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:11.498698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:11.499407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:11.500773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50a17772-3dae-4c3d-86a6-1ff3f11194cb None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:11.507451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 655/1301] 10.222.0.18 () {56 vars in 1087 bytes} [Sun Jul 24 00:51:11 2022] GET /identity/v3/services => generated 2191 bytes in 143 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:51:13.523267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83909e-fa08-4c1e-8065-1f99ef1c75b1 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:13.523797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83909e-fa08-4c1e-8065-1f99ef1c75b1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:13.523891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc83909e-fa08-4c1e-8065-1f99ef1c75b1 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:13.525194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 647/1302] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:51:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:51:13.531365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:13.531756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:13.531918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:13.548330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:51:13.549511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:51:13.613905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 656/1303] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:51:13 2022] POST /identity/v3/auth/tokens => generated 2211 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 364 bytes (1 switches on core 0) Jul 24 00:51:13.636648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc83909e-fa08-4c1e-8065-1f99ef1c75b1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:13.731970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:13.738856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc83909e-fa08-4c1e-8065-1f99ef1c75b1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:13.770907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:13.771700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:13.771987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:13.772265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] PATH_INFO: `/v3/services/placement` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:13.772908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] RBAC: Authorizing `identity:get_service(service_id=placement)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:13.774268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] Could not find service: placement.: keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 152, in get_service Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.driver.get_service(service_id) Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 196, in get_service Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(session, service_id).to_dict() Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/backends/sql.py", line 191, in _get_service Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application During handling of the above exception, another exception occurred: Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Traceback (most recent call last): Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1523, in full_dispatch_request Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application rv = self.dispatch_request() Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/app.py", line 1509, in dispatch_request Jul 24 00:51:13.786525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 467, in wrapper Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = resource(*args, **kwargs) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask/views.py", line 84, in view Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return current_app.ensure_sync(self.dispatch_request)(*args, **kwargs) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/flask_restful/__init__.py", line 582, in dispatch_request Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application resp = meth(*args, **kwargs) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 45, in get Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._get_service(service_id) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/api/services.py", line 34, in _get_service Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.wrap_member(PROVIDERS.catalog_api.get_service(service_id)) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/common/manager.py", line 115, in wrapped Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application __ret_val = __f(*args, **kwargs) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/decorator.py", line 232, in fun Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return caller(func, *(extras + args), **kw) Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1577, in get_or_create_for_user_func Jul 24 00:51:13.787777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.get_or_create( Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 1042, in get_or_create Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application with Lock( Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 185, in __enter__ Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self._enter() Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 94, in _enter Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application generated = self._enter_create(value, createdtime) Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/lock.py", line 178, in _enter_create Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application return self.creator() Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/usr/local/lib/python3.8/dist-packages/dogpile/cache/region.py", line 995, in gen_value Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application created_value = creator( Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application File "/opt/stack/keystone/keystone/catalog/core.py", line 154, in get_service Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application raise exception.ServiceNotFound(service_id=service_id) Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application keystone.exception.ServiceNotFound: Could not find service: placement. Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: ERROR keystone.server.flask.application Jul 24 00:51:13.788778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 648/1304] 10.222.0.18 () {56 vars in 1117 bytes} [Sun Jul 24 00:51:13 2022] GET /identity/v3/services/placement => generated 90 bytes in 152 msecs (HTTP/1.1 404) 5 headers in 183 bytes (1 switches on core 0) Jul 24 00:51:13.793706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:13.867273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:13.891538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893158e8-b9cc-4357-9661-6364bdae6d2b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:13.924208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:13.924860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:13.925141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:13.925414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] PATH_INFO: `/v3/services` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:13.926115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] RBAC: Adding query filter params (name=placement) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:51:13.926419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] RBAC: Authorizing `identity:list_services()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:13.927818 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e14b2ab-54f2-4267-8d56-9eb329020bd9 None admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:13.932792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 657/1305] 10.222.0.18 () {56 vars in 1116 bytes} [Sun Jul 24 00:51:13 2022] GET /identity/v3/services?name=placement => generated 358 bytes in 140 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:51:13.939210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:51:14.034980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa9a8b-d4e1-42e1-a919-14e215b1d35c None admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:51:14.067177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': 'all', 'project_id': None, 'project_domain_id': None, 'roles': ['admin', 'member', 'reader'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:51:14.067810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:51:14.068092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:51:14.068364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] PATH_INFO: `/v3/endpoints` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:51:14.069093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] RBAC: Authorizing `identity:create_endpoint()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:51:14.070440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aaaff012-e4de-4839-b233-e5e9f2d1427a None admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:51:14.088192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 649/1306] 10.222.0.18 () {60 vars in 1144 bytes} [Sun Jul 24 00:51:13 2022] POST /identity/v3/endpoints => generated 331 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:51:23.930231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=30 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:51:24.084437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:16.559757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe534-4669-4716-8666-b67a623cb834 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:16.560337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe534-4669-4716-8666-b67a623cb834 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:16.560337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-345fe534-4669-4716-8666-b67a623cb834 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:16.565687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 658/1307] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:16 2022] GET /identity => generated 270 bytes in 7 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:16.567045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e2be05-5974-4f7b-aaf8-86816c1c5ea9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:16.567487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e2be05-5974-4f7b-aaf8-86816c1c5ea9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:16.567613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64e2be05-5974-4f7b-aaf8-86816c1c5ea9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:16.658966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-64e2be05-5974-4f7b-aaf8-86816c1c5ea9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:16.660395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-64e2be05-5974-4f7b-aaf8-86816c1c5ea9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:16.804940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 650/1308] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 239 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:52:16.870970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:16.871306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:16.872008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:16.876416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 659/1309] 10.222.0.18 () {54 vars in 965 bytes} [Sun Jul 24 00:52:16 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:16.883179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620061-a8e5-4c1f-8fab-0cff2616a0b6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:16.883537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620061-a8e5-4c1f-8fab-0cff2616a0b6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:16.883914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31620061-a8e5-4c1f-8fab-0cff2616a0b6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:16.991628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-31620061-a8e5-4c1f-8fab-0cff2616a0b6 None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:16.993483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-31620061-a8e5-4c1f-8fab-0cff2616a0b6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:17.022808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:17.086998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 651/1310] 10.222.0.18 () {58 vars in 1062 bytes} [Sun Jul 24 00:52:16 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:17.095340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:17.097164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:17.211206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc0eecee-75e6-45f6-95d0-6f59eb178e76 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:17.287584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:17.288235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:17.288515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:17.288785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:17.290406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:17.349306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:17.409641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:17.411136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8355fcb1-c830-4846-b8e2-a3b96001e58c service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:17.430875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 660/1311] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 336 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:52:19.797167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:19.797664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:19.797748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:19.800170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 652/1312] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:19 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:19.804549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340176de-c579-41fd-956e-811ed8d2f698 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:19.805014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340176de-c579-41fd-956e-811ed8d2f698 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:19.805131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-340176de-c579-41fd-956e-811ed8d2f698 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:19.885283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-340176de-c579-41fd-956e-811ed8d2f698 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:19.886973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-340176de-c579-41fd-956e-811ed8d2f698 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:19.992247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 661/1313] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 189 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:20.031985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:20.033551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:20.161635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-870faa17-d86a-46f6-9758-504917c572d0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:20.241528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:20.244770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:20.244968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:20.246418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:20.247346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:20.249266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:20.360685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:20.362106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0cd4244c-59b6-434f-83cd-8e12ab28a3f2 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:20.381975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 653/1314] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 351 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:22.710607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:22.711158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:22.711158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:22.712471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 662/1315] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:22 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:22.718291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000b45b-ce88-43f2-8aa0-7d59838fb9c7 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:22.718720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000b45b-ce88-43f2-8aa0-7d59838fb9c7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:22.718875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8000b45b-ce88-43f2-8aa0-7d59838fb9c7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:22.782209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8000b45b-ce88-43f2-8aa0-7d59838fb9c7 None None] MFA Rules not processed for user `801a75d4248e4a86a827af56143e1da2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:22.783447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8000b45b-ce88-43f2-8aa0-7d59838fb9c7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:22.833666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:22.878132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 654/1316] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:22 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:22.918067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:22.919631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:22.929169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:23.022948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63339568-a27f-4de4-98ea-ac9b4d7165a8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:23.094406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:23.095159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:23.095441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:23.095714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:23.097349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:23.159086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:23.205931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:23.207402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c280de6-8ed8-4d09-a672-2ec8b24c1cd0 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:23.226930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 663/1317] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:22 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:25.795486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e75a94-7eb0-4791-bfb4-4933d487a8b8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:25.795991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e75a94-7eb0-4791-bfb4-4933d487a8b8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:25.795991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e75a94-7eb0-4791-bfb4-4933d487a8b8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:25.797388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 655/1318] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:25.802686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a12d7f-6185-4d90-ad01-e0d6aa09690e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:25.803153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a12d7f-6185-4d90-ad01-e0d6aa09690e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:25.803248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2a12d7f-6185-4d90-ad01-e0d6aa09690e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:25.866182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c2a12d7f-6185-4d90-ad01-e0d6aa09690e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:25.867395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c2a12d7f-6185-4d90-ad01-e0d6aa09690e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:25.963321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 664/1319] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:25 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:26.015281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:26.015567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:26.015844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:26.017211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 656/1320] 10.222.0.18 () {54 vars in 965 bytes} [Sun Jul 24 00:52:26 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:26.022904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66a5cc-ec1a-4094-83d4-34dae3140f2a None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:26.023200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66a5cc-ec1a-4094-83d4-34dae3140f2a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:26.023484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d66a5cc-ec1a-4094-83d4-34dae3140f2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:26.043866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:26.097967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1d66a5cc-ec1a-4094-83d4-34dae3140f2a None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:26.099138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1d66a5cc-ec1a-4094-83d4-34dae3140f2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:26.173708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 665/1321] 10.222.0.18 () {58 vars in 1062 bytes} [Sun Jul 24 00:52:26 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:26.183962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:26.185476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:26.285418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1b0555f-5a0b-496b-a97f-167f8b1633a1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:26.356608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:26.357169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:26.357452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:26.357723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:26.359363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:26.367621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:26.466791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:26.468240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dca8f65-32ad-48c0-94c3-a49b0ff8ba2f service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:26.486153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 657/1322] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:26 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:52:29.608443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:29.608995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:29.608995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:29.610343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 666/1323] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:29 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:29.615528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa04767-09eb-4098-a82e-9c6f92536b4a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:29.615990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa04767-09eb-4098-a82e-9c6f92536b4a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:29.616207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faa04767-09eb-4098-a82e-9c6f92536b4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:29.680174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-faa04767-09eb-4098-a82e-9c6f92536b4a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:29.681374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-faa04767-09eb-4098-a82e-9c6f92536b4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:29.737327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:29.777989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 658/1324] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:29.814937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:29.816588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:29.927799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3bad649-2e83-4234-97a0-0386dec3b37e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:29.941565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:30.006360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:30.007113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:30.007315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:30.007583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:30.009215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:30.117920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:30.119399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52dc09eb-dbfe-43df-818f-82186e4ead42 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:30.137741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 667/1325] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 323 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:32.261740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:32.262248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:32.262248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:32.263626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 659/1326] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:32 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:32.268819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2a0d71-b4c0-4112-ad56-fdf0230a2ef0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:32.269275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2a0d71-b4c0-4112-ad56-fdf0230a2ef0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:32.269389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a2a0d71-b4c0-4112-ad56-fdf0230a2ef0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:32.319478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:32.339050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8a2a0d71-b4c0-4112-ad56-fdf0230a2ef0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:32.340282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8a2a0d71-b4c0-4112-ad56-fdf0230a2ef0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:32.440779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 668/1327] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:32.477351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:32.479488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:32.583050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5f334e1-c00a-41ec-8ce3-7337ada1c23a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:32.656419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:32.657082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:32.657365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:32.657645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:32.659343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:32.678844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:32.777046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:32.778709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d56674f3-a00d-43e0-9381-37c60153fd15 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:32.797519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 660/1328] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:32 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:34.910039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:34.910595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:34.910688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:34.912064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 669/1329] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:34.918017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8911f4-d651-422d-9b05-2e032bf9f690 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:34.918447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8911f4-d651-422d-9b05-2e032bf9f690 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:34.918577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df8911f4-d651-422d-9b05-2e032bf9f690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:34.985580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-df8911f4-d651-422d-9b05-2e032bf9f690 None None] MFA Rules not processed for user `801a75d4248e4a86a827af56143e1da2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:34.986848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-df8911f4-d651-422d-9b05-2e032bf9f690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:35.051538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:35.087981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 661/1330] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:34 2022] POST /identity/v3/auth/tokens => generated 3233 bytes in 171 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:35.127004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:35.128542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:35.237489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7630f2a-e0f5-4e04-a37d-97747c8605e6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:35.257176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:35.313535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:35.314190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:35.314473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:35.314745 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:35.316506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:35.425812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:35.427296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba6ae3e5-a1e9-458c-8c1d-4b4144d169d2 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:35.453698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 670/1331] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:35 2022] GET /identity/v3/auth/tokens => generated 3233 bytes in 327 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:37.745775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:37.746373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:37.746373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:37.747747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 662/1332] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:37 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:37.752989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd656a-8351-4cb7-9395-a58968941432 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:37.753420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd656a-8351-4cb7-9395-a58968941432 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:37.753537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15fd656a-8351-4cb7-9395-a58968941432 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:37.821836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:37.824525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-15fd656a-8351-4cb7-9395-a58968941432 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:37.826384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-15fd656a-8351-4cb7-9395-a58968941432 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:37.920640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 671/1333] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:37.956476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:37.957995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:38.059823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46a54239-a2b7-49d7-9ec3-30ece2241705 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:38.134449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:38.135070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:38.135341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:38.135622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:38.137267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:38.164987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:38.246252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:38.247715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc1d6f36-d6db-4c6f-abb9-13c53c14c89b service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:38.266655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 663/1334] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:40.324687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:40.325280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:40.325280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:40.326595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 672/1335] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:40 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:40.331845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50148927-0f45-4c3e-ad61-4f69598740c1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:40.332293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50148927-0f45-4c3e-ad61-4f69598740c1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:40.332405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50148927-0f45-4c3e-ad61-4f69598740c1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:40.396533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-50148927-0f45-4c3e-ad61-4f69598740c1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:40.397753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-50148927-0f45-4c3e-ad61-4f69598740c1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:40.472898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:40.496892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 664/1336] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:40.536484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:40.538020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:40.641236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95b78b3-82f4-4b02-a60c-90b56e5d9573 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:40.669383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:40.714557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:40.715243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:40.715532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:40.715810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:40.717461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:40.826222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:40.827688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52d1dff6-d37c-4638-a16e-67d5b4eeacaa service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:40.845469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 673/1337] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 309 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:43.284266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:43.284860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:43.284860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:43.286203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 665/1338] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:43.291682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2df7-a9bc-44eb-becb-7a62d29d2378 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:43.292085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2df7-a9bc-44eb-becb-7a62d29d2378 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:43.292247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1fad2df7-a9bc-44eb-becb-7a62d29d2378 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:43.357973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1fad2df7-a9bc-44eb-becb-7a62d29d2378 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:43.359204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1fad2df7-a9bc-44eb-becb-7a62d29d2378 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:43.367242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:43.455392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 674/1339] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:43 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:43.491606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:43.493124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:43.598149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d66cb0cb-29e5-44d9-8780-3a7190944679 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:43.670024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:43.670669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:43.670982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:43.671258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:43.672885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:43.707692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:43.780064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:43.781492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f750d06-1bec-4ab7-9b11-30dd24853e54 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:43.801124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 666/1340] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:43 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:47.831865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:47.832365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:47.832365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:47.833705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 675/1341] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:47.839682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94a68a5-0e38-4ae8-b390-9e7efd7cd6a9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:47.840129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94a68a5-0e38-4ae8-b390-9e7efd7cd6a9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:47.840247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f94a68a5-0e38-4ae8-b390-9e7efd7cd6a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:47.905274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f94a68a5-0e38-4ae8-b390-9e7efd7cd6a9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:47.906500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f94a68a5-0e38-4ae8-b390-9e7efd7cd6a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:47.992958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:48.005620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 667/1342] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:48.046563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:48.048165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:48.154855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9763f98-90c5-4976-8a27-b216bc3166de None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:48.193682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:48.230054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:48.230888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:48.231177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:48.231454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:48.233110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:48.348652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:48.350111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94d4e90c-e384-40a1-8c06-96d9a8c6a805 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:48.369695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 676/1343] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 324 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:50.998932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:50.999508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:50.999508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:51.000875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 668/1344] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:50 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:51.006502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f602c-2fa4-4182-bbdf-413443c1339b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:51.007087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f602c-2fa4-4182-bbdf-413443c1339b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:51.007234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f66f602c-2fa4-4182-bbdf-413443c1339b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:51.077735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f66f602c-2fa4-4182-bbdf-413443c1339b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:51.078982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f66f602c-2fa4-4182-bbdf-413443c1339b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:51.099419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:51.180834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 677/1345] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:51 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:51.219823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:51.221437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:51.324162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a67af6fe-be05-4543-9ebe-7bf656c1e614 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:51.398921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:51.399600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:51.399998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:51.400455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:51.402143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:51.445968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:51.513006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:51.514487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c91b01b3-26ca-4aa6-b675-02d572e1a573 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:51.532467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 669/1346] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:51 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 313 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:52:55.570687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:55.571786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:55.571953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:55.573522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 678/1347] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:52:55 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:52:55.578643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37894bfd-551d-4bad-9186-7c951152c182 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:55.579086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37894bfd-551d-4bad-9186-7c951152c182 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:55.579239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37894bfd-551d-4bad-9186-7c951152c182 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:55.658221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-37894bfd-551d-4bad-9186-7c951152c182 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:52:55.659597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-37894bfd-551d-4bad-9186-7c951152c182 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:55.764529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:55.767446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 670/1348] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:52:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:52:55.807050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:52:55.808870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:55.921550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9dbd094-3212-4c40-838b-2b6c532cfb02 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:52:55.971128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:52:55.999513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:52:56.000170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:52:56.000456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:52:56.000736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:52:56.002408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:52:56.117641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:52:56.119713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ecafa199-db21-4417-b96c-14caa6e5abef service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:52:56.141093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 679/1349] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:52:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 335 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:01.092528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ad9a4-b916-4c5c-9280-8ee5e0df1d04 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:01.092998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ad9a4-b916-4c5c-9280-8ee5e0df1d04 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:01.093089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c8ad9a4-b916-4c5c-9280-8ee5e0df1d04 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:01.097278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 671/1350] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 00:53:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:01.112017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41d75-eb77-4944-bb8c-acd873a9c65f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:01.112444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41d75-eb77-4944-bb8c-acd873a9c65f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:01.112646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cc41d75-eb77-4944-bb8c-acd873a9c65f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:01.237362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1cc41d75-eb77-4944-bb8c-acd873a9c65f None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:01.238609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1cc41d75-eb77-4944-bb8c-acd873a9c65f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:01.258495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:01.365260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 680/1351] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 00:53:01 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 254 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:05.610655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f1c6-6389-4bc3-8dea-4fb885316b5b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:05.611237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f1c6-6389-4bc3-8dea-4fb885316b5b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:05.611350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2923f1c6-6389-4bc3-8dea-4fb885316b5b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:05.612983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 672/1352] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 00:53:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:05.619606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e7163-c3dc-475f-89a7-d5f57b1cd907 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:05.620083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e7163-c3dc-475f-89a7-d5f57b1cd907 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:05.620180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf0e7163-c3dc-475f-89a7-d5f57b1cd907 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:05.701148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bf0e7163-c3dc-475f-89a7-d5f57b1cd907 None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:05.702261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bf0e7163-c3dc-475f-89a7-d5f57b1cd907 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:05.777654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:05.821597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 681/1353] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 00:53:05 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:07.101390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161d5c15-1c4c-4c36-aa21-a7c2488906dc None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:07.102424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161d5c15-1c4c-4c36-aa21-a7c2488906dc None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:07.102590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-161d5c15-1c4c-4c36-aa21-a7c2488906dc None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:07.104000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 673/1354] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 00:53:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:07.109372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf52634-f22f-4da8-9057-8987e07fe64b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:07.109789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf52634-f22f-4da8-9057-8987e07fe64b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:07.110062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf52634-f22f-4da8-9057-8987e07fe64b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:07.187037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-aaf52634-f22f-4da8-9057-8987e07fe64b None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:07.188240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-aaf52634-f22f-4da8-9057-8987e07fe64b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:07.296269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 682/1355] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 00:53:07 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 188 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:53:10.780349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde3db-e7da-4b44-b4ad-639ffa487798 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:10.780893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde3db-e7da-4b44-b4ad-639ffa487798 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:10.780893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dde3db-e7da-4b44-b4ad-639ffa487798 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:10.783067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 674/1356] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:10 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:10.788100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f88d3-772d-4b74-a28c-50e24f2c093d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:10.788588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f88d3-772d-4b74-a28c-50e24f2c093d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:10.788803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4e1f88d3-772d-4b74-a28c-50e24f2c093d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:10.835814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:10.875833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4e1f88d3-772d-4b74-a28c-50e24f2c093d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:10.877294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4e1f88d3-772d-4b74-a28c-50e24f2c093d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:10.987530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 683/1357] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:11.042064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:11.042803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:11.043245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:11.045481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 675/1358] 10.222.0.18 () {54 vars in 962 bytes} [Sun Jul 24 00:53:11 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:11.053557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10d0b46-5ede-4357-8f95-8acfe2e5504d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:11.053999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10d0b46-5ede-4357-8f95-8acfe2e5504d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:11.054385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f10d0b46-5ede-4357-8f95-8acfe2e5504d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:11.138863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f10d0b46-5ede-4357-8f95-8acfe2e5504d None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:11.140241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f10d0b46-5ede-4357-8f95-8acfe2e5504d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:11.194763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:11.267888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 684/1359] 10.222.0.18 () {58 vars in 1059 bytes} [Sun Jul 24 00:53:11 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 215 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:11.275807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:11.277341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:11.443667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2496835-b8c4-41b9-8f22-44438ed3afe0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:11.523268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:11.566996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:11.567644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:11.567930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:11.568208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:11.569812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:11.669331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:11.671064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e73d8ea7-3cf4-47b6-9b48-f4a8d8c68dbb service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:11.693735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 676/1360] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 418 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:13.969690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:13.970240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:13.970240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:13.971747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 685/1361] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:13.976950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281053fb-533f-46ca-a30f-b3f3f601f04e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:13.977415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281053fb-533f-46ca-a30f-b3f3f601f04e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:13.977525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-281053fb-533f-46ca-a30f-b3f3f601f04e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:14.047184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-281053fb-533f-46ca-a30f-b3f3f601f04e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:14.048391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-281053fb-533f-46ca-a30f-b3f3f601f04e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:14.087977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:14.155478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 677/1362] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 180 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:14.201388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:14.203539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:14.384637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36bd5937-7fc0-48bf-bd9f-66b25097305f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:14.400747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:14.533388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:14.534142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:14.534495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:14.534861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:14.536934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:14.631492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:14.633168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d5f483c-f334-4d7b-81bb-ae936ac02ecb service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:14.653176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 686/1363] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:14 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 452 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:17.041175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3f521d-862d-4476-a0fe-046eaf77fd77 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:17.041752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3f521d-862d-4476-a0fe-046eaf77fd77 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:17.041752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d3f521d-862d-4476-a0fe-046eaf77fd77 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:17.047644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 678/1364] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:17 2022] GET /identity => generated 270 bytes in 8 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:17.049478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c8259d-bacf-4b7d-96af-f94dde3da7d5 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:17.049917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c8259d-bacf-4b7d-96af-f94dde3da7d5 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:17.050170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69c8259d-bacf-4b7d-96af-f94dde3da7d5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:17.069392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:17.127191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-69c8259d-bacf-4b7d-96af-f94dde3da7d5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:17.128617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-69c8259d-bacf-4b7d-96af-f94dde3da7d5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:17.231586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 687/1365] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 183 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:17.276283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:17.276690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:17.277206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:17.279510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 679/1366] 10.222.0.18 () {54 vars in 962 bytes} [Sun Jul 24 00:53:17 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:17.284996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438cf4c3-5af2-4d79-ab18-8960550c20ee None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:17.285554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438cf4c3-5af2-4d79-ab18-8960550c20ee None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:17.286045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-438cf4c3-5af2-4d79-ab18-8960550c20ee None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:17.373414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-438cf4c3-5af2-4d79-ab18-8960550c20ee None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:17.374727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-438cf4c3-5af2-4d79-ab18-8960550c20ee None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:17.425417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:17.522341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 688/1367] 10.222.0.18 () {58 vars in 1059 bytes} [Sun Jul 24 00:53:17 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:17.531698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:17.533233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:17.694011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4155de1-eb99-430a-a58b-b7124d16617f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:17.715801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:17.816867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:17.817701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:17.818089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:17.818459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:17.820744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:17.915035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:17.916715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be06d5c-711d-4f6b-863e-89a8679962a6 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:17.935987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 680/1368] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 405 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:18.946340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0695a6e-7493-41f9-9f2e-a02bed685552 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:18.947121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0695a6e-7493-41f9-9f2e-a02bed685552 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:18.947121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0695a6e-7493-41f9-9f2e-a02bed685552 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:18.948483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 689/1369] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:18 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:18.953571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab42b80-8694-457a-80e5-edb8d36c1298 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:18.953983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab42b80-8694-457a-80e5-edb8d36c1298 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:18.954153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab42b80-8694-457a-80e5-edb8d36c1298 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:18.986067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:19.022872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8ab42b80-8694-457a-80e5-edb8d36c1298 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:19.024682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8ab42b80-8694-457a-80e5-edb8d36c1298 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:19.129764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 681/1370] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:19.474331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:19.474621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:19.474926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:19.477010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 690/1371] 10.222.0.18 () {54 vars in 964 bytes} [Sun Jul 24 00:53:19 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:19.482683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b11da-d117-46e0-a618-5e2e4eb9cbb9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:19.483009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b11da-d117-46e0-a618-5e2e4eb9cbb9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:19.483291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c73b11da-d117-46e0-a618-5e2e4eb9cbb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:19.564552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c73b11da-d117-46e0-a618-5e2e4eb9cbb9 None None] MFA Rules not processed for user `5cba3eae172d459bb5c29b712945691c`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:19.565758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c73b11da-d117-46e0-a618-5e2e4eb9cbb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:19.614254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:19.646265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 682/1372] 10.222.0.18 () {58 vars in 1061 bytes} [Sun Jul 24 00:53:19 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:19.653960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:19.655498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:19.769612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bfd09de6-355a-4546-831c-3577220a6285 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:19.815278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:19.853675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:19.854328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:19.854607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:19.854910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:19.856554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:19.971536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:19.973006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0f1ab6eb-c1f7-441a-879a-bfd1f8afaec5 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:19.994640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 691/1373] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 341 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:20.035481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:20.035965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:20.036048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:20.037542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 683/1374] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:20.042676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143662f9-28cf-4b56-a321-fae5a7810470 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:20.043003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143662f9-28cf-4b56-a321-fae5a7810470 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:20.043280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143662f9-28cf-4b56-a321-fae5a7810470 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:20.112343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-143662f9-28cf-4b56-a321-fae5a7810470 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:20.113499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-143662f9-28cf-4b56-a321-fae5a7810470 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:20.136617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:20.238057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 692/1375] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 190 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:20.277615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:20.278656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:20.437134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a40920-0a50-421e-944d-0d654fafa727 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:20.488620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:20.551605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:20.552256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:20.552539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:20.552816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:20.554571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:20.643523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:20.645128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f39a38d7-a05d-4533-89d1-d29580aa2fb5 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:20.664019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 684/1376] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 387 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:22.727480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:22.727995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:22.727995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:22.729306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 693/1377] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:22 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:22.734450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f0c84-16d9-40dc-8cd3-6f9b5a4a6479 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:22.734905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f0c84-16d9-40dc-8cd3-6f9b5a4a6479 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:22.735044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d90f0c84-16d9-40dc-8cd3-6f9b5a4a6479 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:22.800958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d90f0c84-16d9-40dc-8cd3-6f9b5a4a6479 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:22.802161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d90f0c84-16d9-40dc-8cd3-6f9b5a4a6479 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:22.810629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:22.907843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 685/1378] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:22.953717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:22.955278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:23.120553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-059a1b74-4d6d-43df-9fea-304369939443 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:23.124088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:23.239865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:23.240515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:23.240796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:23.241069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:23.243362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:23.334634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:23.336381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-650cd7bb-a316-412f-b345-8225b6a996b6 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:23.361346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 694/1379] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 408 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:25.472478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:25.473063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:25.473263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:25.474915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 686/1380] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:53:25.480955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dbbf41-cd6c-4941-896a-4ee19397e5dd None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:25.481450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dbbf41-cd6c-4941-896a-4ee19397e5dd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:25.481544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2dbbf41-cd6c-4941-896a-4ee19397e5dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:25.491787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:25.558548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e2dbbf41-cd6c-4941-896a-4ee19397e5dd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:25.559796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e2dbbf41-cd6c-4941-896a-4ee19397e5dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:25.660976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 695/1381] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:25.704859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:25.706536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:25.859311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:25.871498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02543019-9196-4662-9c83-69a19a4bb6df None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:25.992975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:25.993635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:25.993991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:25.994341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:25.996084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:26.090930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:26.092635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f477fa3d-0433-4017-b10f-033ddf11d9a3 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:26.104222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:26.113694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 687/1382] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 409 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:28.223120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:28.223662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:28.223921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:28.226477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 696/1383] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:28 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:28.232029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d111-d235-4df0-8242-e2c4ad2b0d36 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:28.232491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d111-d235-4df0-8242-e2c4ad2b0d36 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:28.232589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ec38d111-d235-4df0-8242-e2c4ad2b0d36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:28.301100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ec38d111-d235-4df0-8242-e2c4ad2b0d36 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:28.302481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ec38d111-d235-4df0-8242-e2c4ad2b0d36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:28.410418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 688/1384] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:28.455516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:28.457150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:28.564739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:28.629909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a976c04b-baa9-4aaa-98aa-dc2226c1a031 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:28.784044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:28.784622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:28.784903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:28.785177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:28.786896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:28.845342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:28.894441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:28.896102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-748cf1e3-689d-4954-b1bc-dc80463ac17c service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:28.915666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 697/1385] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 461 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:31.094644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:31.095207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:31.095207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:31.096532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 689/1386] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:31 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:53:31.102177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633f80-2e3f-44cf-b881-28f0288a29b1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:31.102950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633f80-2e3f-44cf-b881-28f0288a29b1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:31.102950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a633f80-2e3f-44cf-b881-28f0288a29b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:31.172910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8a633f80-2e3f-44cf-b881-28f0288a29b1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:31.174137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8a633f80-2e3f-44cf-b881-28f0288a29b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:31.273523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 698/1387] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:31.326812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:31.328961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:31.424008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:31.509830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0190fac1-df9e-400c-bba9-c31a27e80489 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:31.635407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:31.636066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:31.636354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:31.636633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:31.638328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:31.678283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:31.749720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:31.751924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef7fcba-d8c3-456c-8561-e1738ba96d46 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:31.775725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 690/1388] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 450 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:34.168013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:34.168677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:34.168677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:34.170041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 699/1389] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:34 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:34.175469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebef06e-3bd1-47b8-bde1-dc37e0f468f1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:34.175947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebef06e-3bd1-47b8-bde1-dc37e0f468f1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:34.176061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ebef06e-3bd1-47b8-bde1-dc37e0f468f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:34.240988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-9ebef06e-3bd1-47b8-bde1-dc37e0f468f1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:34.242231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9ebef06e-3bd1-47b8-bde1-dc37e0f468f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:34.347356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 691/1390] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:34.392191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:34.397218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:34.429831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:34.548363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b4e41c1-5f5c-4141-9206-40e9ac4cba64 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:34.649930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:34.662205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:34.662906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:34.663193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:34.663505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:34.665234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:34.753073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:34.754605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13cb4ad4-7ca1-4140-a6cc-241529e5a854 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:34.773770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 700/1391] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 382 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:35.220617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceddced9-fe5a-47bd-86bb-02689da2acb7 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:35.221201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceddced9-fe5a-47bd-86bb-02689da2acb7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:35.221201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceddced9-fe5a-47bd-86bb-02689da2acb7 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:35.222568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 692/1392] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:35.228636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1db8c-abc2-496c-a52d-58130f5c99ef None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:35.229080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1db8c-abc2-496c-a52d-58130f5c99ef None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:35.229195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1db8c-abc2-496c-a52d-58130f5c99ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:35.296011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cee1db8c-abc2-496c-a52d-58130f5c99ef None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:35.297210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cee1db8c-abc2-496c-a52d-58130f5c99ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:35.355911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:35.399332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 701/1393] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:35.456799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:35.457110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:35.457392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:35.458825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 693/1394] 10.222.0.18 () {54 vars in 966 bytes} [Sun Jul 24 00:53:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:35.463924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a16e7-a28d-4fea-8bd3-3bfd68483172 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:35.464214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a16e7-a28d-4fea-8bd3-3bfd68483172 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:35.464491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a5a16e7-a28d-4fea-8bd3-3bfd68483172 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:35.540244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5a5a16e7-a28d-4fea-8bd3-3bfd68483172 None None] MFA Rules not processed for user `e85523ac61714842a523a22a7bc2888b`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:35.541360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5a5a16e7-a28d-4fea-8bd3-3bfd68483172 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:35.619227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 702/1395] 10.222.0.18 () {58 vars in 1063 bytes} [Sun Jul 24 00:53:35 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:53:35.626681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:35.628274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:35.641766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:35.734509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9751d2fa-5e61-41a6-b718-a4aee4169752 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:35.807299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:35.807876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:35.808197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:35.808476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:35.810125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:35.876699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:35.919527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:35.920994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ba020e6-1d58-48cc-9a0d-d52898b0dbf6 service glance] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:35.941112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 694/1396] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 315 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:36.659895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e1100-b3f3-492e-808d-26fe211bf8b3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:36.660357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e1100-b3f3-492e-808d-26fe211bf8b3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:36.660454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a6e1100-b3f3-492e-808d-26fe211bf8b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:36.727145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6a6e1100-b3f3-492e-808d-26fe211bf8b3 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:36.728405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6a6e1100-b3f3-492e-808d-26fe211bf8b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:36.736842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:36.814226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 703/1397] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 155 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:36.863680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22af41b0-842f-481b-bf15-d3a6b7fdf1fd None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:36.864130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22af41b0-842f-481b-bf15-d3a6b7fdf1fd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:36.864301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22af41b0-842f-481b-bf15-d3a6b7fdf1fd None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:36.865871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 695/1398] 10.222.0.18 () {54 vars in 943 bytes} [Sun Jul 24 00:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:36.872348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:36.872636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:36.872914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:36.876856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9865eb-c028-4418-95c1-1f72774af01f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:36.877240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9865eb-c028-4418-95c1-1f72774af01f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:36.877528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa9865eb-c028-4418-95c1-1f72774af01f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:36.878858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 696/1399] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:53:36.884330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:36.884626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:36.884909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:36.952487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:36.954040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:36.954837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] MFA Rules not processed for user `d1387a119eb04d0a9efe0164de9a8c3c`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:36.956089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:37.043935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:37.055609 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 697/1400] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:53:37.068293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 704/1401] 10.222.0.18 () {58 vars in 1040 bytes} [Sun Jul 24 00:53:36 2022] POST /identity/v3/auth/tokens => generated 3294 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:53:37.078386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:37.079665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:37.101568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:37.102842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:37.107873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:37.187092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df5f84c-e191-4509-af70-92d0080be656 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:37.203440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b5c52f9-2920-4e0f-afd1-e59594b81b0b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:37.293350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:37.293998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:37.294282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:37.294558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:37.296295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:37.312584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:37.320538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:37.321195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:37.321606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:37.321885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:37.323584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:37.339085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:37.381429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:37.383628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-994381ef-b521-410e-97a9-db4837fd8655 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:37.391697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 698/1402] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 00:53:37 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 00:53:37.425161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:37.426938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a006e80-9951-471d-bc19-3c7534720394 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:37.446582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 705/1403] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:37 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 346 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:53:39.621156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:39.621741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:39.621741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:39.625064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 699/1404] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:39 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:39.628373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792eb76-5ee2-4e8b-aac3-e7ecf48ebab0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:39.628794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792eb76-5ee2-4e8b-aac3-e7ecf48ebab0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:39.628946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a792eb76-5ee2-4e8b-aac3-e7ecf48ebab0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:39.711973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a792eb76-5ee2-4e8b-aac3-e7ecf48ebab0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:39.713555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a792eb76-5ee2-4e8b-aac3-e7ecf48ebab0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:39.783034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:39.823014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 706/1405] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 195 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:39.867936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:39.869474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:40.020935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14291b2d-01af-4906-bdd8-8bdf21ecfe2b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:40.061314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:40.144340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:40.145309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:40.145770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:40.146229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:40.148712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:40.256349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:40.258592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c2ad1bc-c396-499c-af9e-e9fc4dc39ca2 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:40.278220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 700/1406] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 411 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:53:42.360474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:42.361032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:42.361032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:42.362401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 707/1407] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:53:42 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:42.368403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c52f0-9b7d-448b-8283-5b8714354c5e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:42.372659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c52f0-9b7d-448b-8283-5b8714354c5e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:42.373064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-516c52f0-9b7d-448b-8283-5b8714354c5e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:42.431963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:42.453913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-516c52f0-9b7d-448b-8283-5b8714354c5e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:42.456144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-516c52f0-9b7d-448b-8283-5b8714354c5e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:42.598201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 701/1408] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:53:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 231 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:42.642242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:53:42.643796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:42.815992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d0444eb-5c70-46dd-89c6-dce00f85778d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:53:42.876187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:42.947088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:53:42.948100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:42.948533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:42.948977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:42.950624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:43.051349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:53:43.053118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5a4c26e-1511-499b-b2e5-271a086b19ef service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:53:43.078964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 708/1409] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:53:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 437 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:53:52.596175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=47 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:53.071451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=75 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:59.671119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074bc99a-91ae-4bff-b8c4-7efad4fc06c8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:59.671653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074bc99a-91ae-4bff-b8c4-7efad4fc06c8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:59.671653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-074bc99a-91ae-4bff-b8c4-7efad4fc06c8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:59.673058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 702/1410] 10.222.0.18 () {54 vars in 918 bytes} [Sun Jul 24 00:53:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:59.678055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76e706a-5b78-42a1-9f75-36c2b1c6750d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:59.678489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76e706a-5b78-42a1-9f75-36c2b1c6750d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:59.678596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c76e706a-5b78-42a1-9f75-36c2b1c6750d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:59.752289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c76e706a-5b78-42a1-9f75-36c2b1c6750d None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:59.753511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c76e706a-5b78-42a1-9f75-36c2b1c6750d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:53:59.858278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 709/1411] 10.222.0.18 () {58 vars in 1015 bytes} [Sun Jul 24 00:53:59 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 181 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:53:59.873186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded3d29-82ed-42f6-97dd-4a4b5489338b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:59.873479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded3d29-82ed-42f6-97dd-4a4b5489338b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:59.873760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded3d29-82ed-42f6-97dd-4a4b5489338b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:59.875170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 703/1412] 10.222.0.18 () {54 vars in 918 bytes} [Sun Jul 24 00:53:59 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:53:59.879774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb982c1-7edd-43b3-af74-9ba21f2a3b8b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:53:59.880067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb982c1-7edd-43b3-af74-9ba21f2a3b8b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:53:59.880349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb982c1-7edd-43b3-af74-9ba21f2a3b8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:53:59.952624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:53:59.955010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1cb982c1-7edd-43b3-af74-9ba21f2a3b8b None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:53:59.956783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1cb982c1-7edd-43b3-af74-9ba21f2a3b8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:00.070494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 710/1413] 10.222.0.18 () {58 vars in 1015 bytes} [Sun Jul 24 00:53:59 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 192 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:02.577703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:02.578213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:02.578318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:02.579662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 704/1414] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:02 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:02.584850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6892b-7500-42ec-864d-ec8884146f06 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:02.585267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6892b-7500-42ec-864d-ec8884146f06 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:02.585411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cce6892b-7500-42ec-864d-ec8884146f06 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:02.651036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cce6892b-7500-42ec-864d-ec8884146f06 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:02.652228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cce6892b-7500-42ec-864d-ec8884146f06 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:02.715052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:02.757425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 711/1415] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:02.799619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:02.801162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:02.944390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40cf4ccf-795e-4bed-8ac5-ffc0df01ac23 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:03.015519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:03.055001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:03.055642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:03.055925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:03.056202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:03.057840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:03.145124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:03.146736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05b0fc06-dc76-44a1-b337-07f6bfa91ce5 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:03.165007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 705/1416] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 366 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:04.062294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:04.062880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:04.063020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:04.064285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 712/1417] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:04.069217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c80a-5c06-43e2-abec-7a32c5b5fa36 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:04.069603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c80a-5c06-43e2-abec-7a32c5b5fa36 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:04.069774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0824c80a-5c06-43e2-abec-7a32c5b5fa36 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:04.143870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0824c80a-5c06-43e2-abec-7a32c5b5fa36 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:04.145072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0824c80a-5c06-43e2-abec-7a32c5b5fa36 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:04.171135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:04.240464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 706/1418] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:04.391372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:04.392839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:04.537510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aed65e62-b444-41c1-8e9e-5fb835840c1d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:04.579613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:04.651385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:04.651954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:04.652234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:04.652508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:04.654155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:04.741204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:04.743448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-062929da-7ae9-460a-b131-bc9fa29566cc service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:04.760941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 713/1419] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:04 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:05.635474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:05.636064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:05.636064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:05.637418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 707/1420] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:05 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:05.642345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e92c6e-f551-4000-bd33-8c093590ad98 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:05.642809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e92c6e-f551-4000-bd33-8c093590ad98 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:05.642963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8e92c6e-f551-4000-bd33-8c093590ad98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:05.696621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:05.719363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e8e92c6e-f551-4000-bd33-8c093590ad98 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:05.720572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e8e92c6e-f551-4000-bd33-8c093590ad98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:05.814514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 714/1421] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:05.965305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:05.966670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:06.109127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8159f762-6bb4-4b0a-99b5-0e02d1270421 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:06.121336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:06.223786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:06.224350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:06.224630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:06.224902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:06.226541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:06.321408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:06.323115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c506a56a-b700-4769-bd7b-8a85c0117701 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:06.342156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 708/1422] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 378 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:07.203871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:07.204412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:07.204504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:07.205925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 715/1423] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:07.212066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539db79-bedb-4c84-882f-8d1bab41a1a1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:07.212503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539db79-bedb-4c84-882f-8d1bab41a1a1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:07.212633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4539db79-bedb-4c84-882f-8d1bab41a1a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:07.231737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:07.287005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4539db79-bedb-4c84-882f-8d1bab41a1a1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:07.288206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4539db79-bedb-4c84-882f-8d1bab41a1a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:07.385151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 709/1424] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:07.537518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:07.539515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:07.648124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:07.685333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4addd9e3-1e4a-41d7-a8d4-c1a6f842f09b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:07.796341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:07.796911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:07.797188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:07.797461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:07.799132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:07.873890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:07.887830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:07.889448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b332614-7b54-4812-9687-28449e265a2e service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:07.907899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 716/1425] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:08.776125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:08.776707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:08.776707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:08.778018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 710/1426] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:54:08.782908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc608154-bcd7-485f-b153-21a248482d9e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:08.783334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc608154-bcd7-485f-b153-21a248482d9e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:08.783471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc608154-bcd7-485f-b153-21a248482d9e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:08.857474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cc608154-bcd7-485f-b153-21a248482d9e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:08.858675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cc608154-bcd7-485f-b153-21a248482d9e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:08.953426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 717/1427] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:09.106605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:09.108109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:09.191934 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:09.253870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b18eae4d-9988-4e3b-aeea-3e0eb88b66f0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:09.364409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:09.364982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:09.365268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:09.365548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:09.367725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:09.415242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:09.456957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:09.458574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-655d5d04-3d61-4edf-9c0a-4493fb02af84 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:09.476184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 711/1428] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:10.278186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:10.278737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:10.278737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:10.280141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 718/1429] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:10 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:10.284916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91871f5-02df-4176-a717-9b6485ce463a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:10.285317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91871f5-02df-4176-a717-9b6485ce463a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:10.285470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e91871f5-02df-4176-a717-9b6485ce463a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:10.359406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e91871f5-02df-4176-a717-9b6485ce463a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:10.360614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e91871f5-02df-4176-a717-9b6485ce463a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:10.457722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 712/1430] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:10.607716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:10.609313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:10.657897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:10.754969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1109d369-9287-438b-a2cc-1bf5832c19f2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:10.866424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:10.867033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:10.867980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:10.868260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:10.869920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:10.876842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:10.958703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:10.960328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86502b47-100f-4bec-aead-c06db17101b8 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:10.978957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 719/1431] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:11.782287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:11.782853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:11.782853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:11.784159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 713/1432] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:11 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:11.789028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a38648a-984b-4511-9be9-84d4c83a6409 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:11.790449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a38648a-984b-4511-9be9-84d4c83a6409 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:11.790449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a38648a-984b-4511-9be9-84d4c83a6409 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:11.865123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3a38648a-984b-4511-9be9-84d4c83a6409 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:11.866333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3a38648a-984b-4511-9be9-84d4c83a6409 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:11.932195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:11.961746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 720/1433] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:12.112202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:12.113684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:12.131495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:12.258897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68063f36-746c-48ef-836f-66183093fbf9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:12.345032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:12.371695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:12.372277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:12.372562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:12.372841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:12.374493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:12.464780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:12.466392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba3406d-4202-4d15-9f0f-c11f178e49d7 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:12.484147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 714/1434] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:13.284756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:13.285340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:13.285340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:13.286683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 721/1435] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:13 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:13.291878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70803c-f5c2-48f0-a9cd-1f53b128382b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:13.292310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70803c-f5c2-48f0-a9cd-1f53b128382b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:13.292428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac70803c-f5c2-48f0-a9cd-1f53b128382b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:13.365252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ac70803c-f5c2-48f0-a9cd-1f53b128382b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:13.366435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ac70803c-f5c2-48f0-a9cd-1f53b128382b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:13.409575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:13.466094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 715/1436] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 175 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:54:13.624016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:13.625461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:13.770696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b050345-e7b0-49f9-9ba8-d00af33690a2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:13.824510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:13.883684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:13.884346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:13.884627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:13.884898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:13.886537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:13.974215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:13.975839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a95a48d4-28db-4585-861c-a620b7058de9 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:13.994463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 722/1437] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:13 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 371 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:14.820045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:14.821114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:14.821114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:14.822513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 716/1438] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:54:14 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:14.827589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c014-838e-4220-832d-bf158b5416cb None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:14.828027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c014-838e-4220-832d-bf158b5416cb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:14.828138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8273c014-838e-4220-832d-bf158b5416cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:14.902572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8273c014-838e-4220-832d-bf158b5416cb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:14.903886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8273c014-838e-4220-832d-bf158b5416cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:14.905904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:14.999983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 723/1439] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:54:14 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:54:15.149756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:15.151238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:15.296445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b587b3b-663b-4b11-a576-9388c9e9fd6b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:15.323280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:15.410046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:15.410702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:15.411035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:15.411319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:15.412971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:15.501651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:15.503286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e820c59-2cc5-4c58-be61-e2f8e5b1ff71 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:15.520918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 717/1440] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:54:17.454077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:17.454560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:17.454657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:17.456045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 724/1441] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:17 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:17.461789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e8a3ab-582a-4e2b-be39-a0f32a3d3f89 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:17.462198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e8a3ab-582a-4e2b-be39-a0f32a3d3f89 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:17.462351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-67e8a3ab-582a-4e2b-be39-a0f32a3d3f89 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:17.502184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:17.528044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-67e8a3ab-582a-4e2b-be39-a0f32a3d3f89 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:17.529232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-67e8a3ab-582a-4e2b-be39-a0f32a3d3f89 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:17.626485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 718/1442] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:17.665863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:17.667485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:17.770528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b1b93f9-d8bb-4142-a0fc-a04d6e8ea0cb None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:17.803395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:17.845132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:17.845780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:17.846071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:17.846350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:17.848028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:17.955851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:17.957209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-64ed7f14-6e8f-4981-99b4-7fcf9a296d4c service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:17.975349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 725/1443] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:54:20.161842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:20.162496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:20.162496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:20.163837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 719/1444] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:20 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:20.169740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830f187-7aec-44a0-aa7f-d4bc71c12516 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:20.170200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830f187-7aec-44a0-aa7f-d4bc71c12516 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:20.170417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3830f187-7aec-44a0-aa7f-d4bc71c12516 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:20.235394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3830f187-7aec-44a0-aa7f-d4bc71c12516 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:20.236616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3830f187-7aec-44a0-aa7f-d4bc71c12516 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:20.249566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:20.334388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 726/1445] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:20.372943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:20.374541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:20.477894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ab0cabd-531d-4473-96f6-8da09766fc8c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:20.489711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:20.550252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:20.550925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:20.551210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:20.551486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:20.553113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:20.666159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:20.667675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbd74da1-4de6-4188-9321-44d6dfd69349 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:20.686043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 720/1446] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:20 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:23.029695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:23.030257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:23.030257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:23.031741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 727/1447] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:23 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:23.036916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e9793-582c-4013-8d59-6dc959a5cd3a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:23.037327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e9793-582c-4013-8d59-6dc959a5cd3a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:23.037479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-035e9793-582c-4013-8d59-6dc959a5cd3a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:23.087662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:23.106004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-035e9793-582c-4013-8d59-6dc959a5cd3a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:23.107244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-035e9793-582c-4013-8d59-6dc959a5cd3a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:23.202512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 721/1448] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:23.241435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:23.243068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:23.345637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbefa509-e4c6-43fc-a3de-93b7045740d1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:23.385486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:23.419807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:23.420454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:23.420734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:23.421005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:23.422640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:23.529168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:23.530603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d974d79c-738c-4607-b4d3-a29121df5165 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:23.548075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 728/1449] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:25.697369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:25.697880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:25.697880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:25.699285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 722/1450] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:25.705214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca966e-131c-4cca-9da1-dde78d71ac8c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:25.705654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca966e-131c-4cca-9da1-dde78d71ac8c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:25.705764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ca966e-131c-4cca-9da1-dde78d71ac8c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:25.769209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-14ca966e-131c-4cca-9da1-dde78d71ac8c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:25.770414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-14ca966e-131c-4cca-9da1-dde78d71ac8c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:25.791032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:25.866242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 729/1451] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:25.906919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:25.908763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:26.014083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48405051-7bb7-4ef8-a6a8-a152f8d37cb8 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:26.033497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:26.088117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:26.088758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:26.089038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:26.089315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:26.090976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:26.198377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:26.199842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7f2fb525-ff92-468c-a987-ee0386317911 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:26.217214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 723/1452] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 311 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:28.306583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:28.307126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:28.307222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:28.308612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 730/1453] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:28 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:28.314303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7552c04-e62a-4006-9177-915cf1b05504 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:28.314788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7552c04-e62a-4006-9177-915cf1b05504 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:28.314888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7552c04-e62a-4006-9177-915cf1b05504 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:28.376835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:28.379661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d7552c04-e62a-4006-9177-915cf1b05504 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:28.381517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d7552c04-e62a-4006-9177-915cf1b05504 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:28.482118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 724/1454] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:28.520407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:28.521979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:28.628017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38ac6b02-1a0a-487f-9820-0ebc467ccc6d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:28.684436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:28.709951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:28.710593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:28.710907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:28.711180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:28.713817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:28.821162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:28.822654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c600f6-8c9e-4656-9a3c-dbd0c4a3a953 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:28.841184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 731/1455] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:28 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 321 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:30.950648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:30.951260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:30.951260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:30.952670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 725/1456] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:30 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:30.958039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316ca-35db-4e91-bbec-5662947d9c51 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:30.958468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316ca-35db-4e91-bbec-5662947d9c51 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:30.958604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13f316ca-35db-4e91-bbec-5662947d9c51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:31.026012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-13f316ca-35db-4e91-bbec-5662947d9c51 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:31.027271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-13f316ca-35db-4e91-bbec-5662947d9c51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:31.057726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:31.129131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 732/1457] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:31.167785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:31.169341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:31.278642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4122bcd-94bb-40ef-a81d-d30b3d3885a5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:31.306826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:31.352888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:31.353546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:31.353833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:31.354113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:31.355846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:31.467027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:31.468469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10a0e973-fbea-4a1d-bc3f-ff09204d4370 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:31.486631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 726/1458] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 319 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:33.573900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:33.574458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:33.574458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:33.575913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 733/1459] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:33.581149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ce1d-796f-4200-8152-5472cdc445db None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:33.581540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ce1d-796f-4200-8152-5472cdc445db None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:33.581713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a954ce1d-796f-4200-8152-5472cdc445db None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:33.645512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a954ce1d-796f-4200-8152-5472cdc445db None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:33.646728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a954ce1d-796f-4200-8152-5472cdc445db None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:33.655548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:33.743677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 727/1460] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:33.781827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:33.783390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:33.886194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18e25c19-e653-4ab9-9db4-63d781d0148b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:33.940360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:33.958978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:33.959675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:33.959956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:33.960228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:33.962432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:34.069154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:34.070506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c94facc-ec44-4d20-ba48-4f019707d87c service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:34.088369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 734/1461] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:33 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:36.175786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:36.176332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:36.176458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:36.177770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 728/1462] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:36 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:36.183012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b2f56b-0d0e-4341-aeaa-14406320ebdf None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:36.183445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b2f56b-0d0e-4341-aeaa-14406320ebdf None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:36.183572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78b2f56b-0d0e-4341-aeaa-14406320ebdf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:36.247435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-78b2f56b-0d0e-4341-aeaa-14406320ebdf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:36.248642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-78b2f56b-0d0e-4341-aeaa-14406320ebdf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:36.288142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:36.347109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 735/1463] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:36.386295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:36.387850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:36.491863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb2393eb-60c6-4837-a802-399e7026f312 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:36.529309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:36.564745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:36.565392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:36.565679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:36.565958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:36.567647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:36.676225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:36.677657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cd05b89-7298-477d-97c4-d8dcd505e3f7 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:36.697293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 729/1464] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 312 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:38.789555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:38.790615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:38.790725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:38.792096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 736/1465] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:38 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:38.797371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb779fa-65bc-423b-a6fa-40860ffb032a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:38.797761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb779fa-65bc-423b-a6fa-40860ffb032a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:38.797930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb779fa-65bc-423b-a6fa-40860ffb032a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:38.861805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6bb779fa-65bc-423b-a6fa-40860ffb032a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:38.862989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6bb779fa-65bc-423b-a6fa-40860ffb032a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:38.880484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:38.958158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 730/1466] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 162 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:38.996154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:38.997687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:39.100067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51988b74-13d5-4a9a-bb41-4f8811024dfe None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:39.165107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:39.172355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:39.172999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:39.173277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:39.173576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:39.175225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:39.281882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:39.283365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b70b877-03a3-4599-b885-b4264cf93cb8 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:39.300769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 737/1467] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 305 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:41.362335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:41.362939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:41.362939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:41.364322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 731/1468] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:54:41 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:54:41.369975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547340b5-f9d8-449b-829d-185c5b3a6dde None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:41.370431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547340b5-f9d8-449b-829d-185c5b3a6dde None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:41.370537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-547340b5-f9d8-449b-829d-185c5b3a6dde None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:41.434623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-547340b5-f9d8-449b-829d-185c5b3a6dde None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:54:41.436135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-547340b5-f9d8-449b-829d-185c5b3a6dde None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:41.484154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:41.532027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 738/1469] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:54:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:54:41.569991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:54:41.571600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:41.675493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6290b6e-3d6e-4172-8110-133c622e981b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:54:41.721956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:41.748245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:54:41.748899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:54:41.749185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:54:41.749468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:54:41.751149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:54:41.857438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:54:41.858898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6ed34e9-aa53-458f-a3ca-b52d040d2a22 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:54:41.877141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 732/1470] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:54:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 308 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:54:51.530152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=22 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:54:51.874978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=66 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:01.666320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264113b1-a168-43d3-a722-d52a103f358d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:01.666918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264113b1-a168-43d3-a722-d52a103f358d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:01.666918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-264113b1-a168-43d3-a722-d52a103f358d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:01.668289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 739/1471] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:01 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:01.673531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1bee-6141-4514-ac33-2c46ec131f8b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:01.674007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1bee-6141-4514-ac33-2c46ec131f8b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:01.674182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-659f1bee-6141-4514-ac33-2c46ec131f8b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:01.739025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-659f1bee-6141-4514-ac33-2c46ec131f8b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:01.740226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-659f1bee-6141-4514-ac33-2c46ec131f8b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:01.835781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 733/1472] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 163 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:01.921178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:01.921404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:01.921680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:01.923088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 740/1473] 10.222.0.18 () {54 vars in 966 bytes} [Sun Jul 24 00:55:01 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:01.928037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db0e23-68dd-4698-a4c0-e495080c4239 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:01.928331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db0e23-68dd-4698-a4c0-e495080c4239 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:01.928617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3db0e23-68dd-4698-a4c0-e495080c4239 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:02.002553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d3db0e23-68dd-4698-a4c0-e495080c4239 None None] MFA Rules not processed for user `e85523ac61714842a523a22a7bc2888b`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:02.003785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d3db0e23-68dd-4698-a4c0-e495080c4239 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:02.024253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:02.080907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 734/1474] 10.222.0.18 () {58 vars in 1063 bytes} [Sun Jul 24 00:55:01 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 154 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:02.088698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:02.090285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:02.193060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-972a3276-d824-4f89-876c-6d7b80f6daef None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:02.265810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:02.266458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:02.266741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:02.267055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:02.268739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:02.322816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:02.377349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:02.378817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0581418a-e106-46f6-ac7e-d4fe90f595c2 service glance] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:02.397844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 741/1475] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:04.530014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:04.530578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:04.530578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:04.532007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 735/1476] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:04 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:04.537273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7889edda-c83c-44f8-bc20-9d1b557afcb0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:04.537729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7889edda-c83c-44f8-bc20-9d1b557afcb0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:04.537837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7889edda-c83c-44f8-bc20-9d1b557afcb0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:04.602878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7889edda-c83c-44f8-bc20-9d1b557afcb0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:04.604095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7889edda-c83c-44f8-bc20-9d1b557afcb0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:04.701108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 742/1477] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:04.724838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:04.726378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:04.852031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1edc6b10-6dd3-4d12-9bcb-e567708ae8e5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:04.901027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:04.947151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:04.948614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:04.948899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:04.949180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:04.949907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] RBAC: Authorizing `identity:list_projects()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:04.951773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:55:04.951773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 00:55:04.951773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 00:55:04.952908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a7036017-c9b6-40b1-bf96-7f32d0437950 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:04.972403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 736/1478] 10.222.0.18 () {56 vars in 1108 bytes} [Sun Jul 24 00:55:04 2022] GET /identity/v3/projects => generated 2553 bytes in 248 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:55:07.068181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:07.068738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:07.068738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:07.070104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 743/1479] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:07 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:07.075460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384be645-0e18-45e5-b942-2f47ef23208e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:07.075917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384be645-0e18-45e5-b942-2f47ef23208e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:07.076028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-384be645-0e18-45e5-b942-2f47ef23208e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:07.149807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-384be645-0e18-45e5-b942-2f47ef23208e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:07.151585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-384be645-0e18-45e5-b942-2f47ef23208e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:07.247025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 737/1480] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 173 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:07.290176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:07.292319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:07.301910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:07.437434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5799bbda-ee0d-401a-af8d-6cdb844d0dd8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:07.516484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:07.548979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:07.549620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:07.549902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:07.550195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:07.551872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:07.639441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:07.641036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19326754-5da1-47d5-b13a-1f782027ceb1 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:07.659288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 744/1481] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 370 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:09.890726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:09.891311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:09.891311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:09.892626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 738/1482] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:09 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:09.897738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d01e9-fd5e-434f-930c-fd368d997486 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:09.898193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d01e9-fd5e-434f-930c-fd368d997486 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:09.898412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b4d01e9-fd5e-434f-930c-fd368d997486 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:09.963881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1b4d01e9-fd5e-434f-930c-fd368d997486 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:09.965090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1b4d01e9-fd5e-434f-930c-fd368d997486 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:10.010683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:10.061256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 745/1483] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:10.097781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:10.099399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:10.123293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:10.202964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07644b49-9245-468e-8566-2a4a9eb36e98 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:10.274752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:10.275426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:10.276212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:10.276486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:10.278146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:10.352133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:10.386678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:10.388143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5573e8c-e98d-4f48-b903-afc4224f569b service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:10.407056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 739/1484] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 310 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:12.504995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:12.505581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:12.505581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:12.506949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 746/1485] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:12 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:12.512841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311ccb-7bdd-4187-997a-bdc67e08a80a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:12.513239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311ccb-7bdd-4187-997a-bdc67e08a80a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:12.513404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c311ccb-7bdd-4187-997a-bdc67e08a80a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:12.702189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7c311ccb-7bdd-4187-997a-bdc67e08a80a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:12.703456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7c311ccb-7bdd-4187-997a-bdc67e08a80a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:12.799203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 740/1486] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 287 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:55:12.837866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:12.840117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:12.948665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64fd2bfb-7687-4996-92ab-05e00d04a871 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:13.019300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:13.021391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:13.022138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:13.022649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:13.023141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:13.025054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:13.133555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:13.135709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63c61514-bb51-4d45-a740-6b3f9e320b6f service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:13.153847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 747/1487] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:15.376554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:15.377119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:15.377119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:15.378512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 741/1488] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:15 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:15.383857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170f83a5-8c58-4533-bda1-1dde79cb840f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:15.384266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170f83a5-8c58-4533-bda1-1dde79cb840f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:15.384422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-170f83a5-8c58-4533-bda1-1dde79cb840f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:15.450886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-170f83a5-8c58-4533-bda1-1dde79cb840f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:15.452126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-170f83a5-8c58-4533-bda1-1dde79cb840f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:15.503820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:15.548765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 748/1489] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:15.589768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:15.591322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:15.633651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:15.700772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c03219aa-c76c-4e6e-8405-9a0432eeb49e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:15.771812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:15.772388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:15.772672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:15.772949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:15.774569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:15.855410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:15.885596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:15.887067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5470922-a157-446f-a3aa-9700a0bef62b service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:15.905040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 742/1490] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 316 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:18.127894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:18.128434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:18.128434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:18.129904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 749/1491] 10.222.0.18 () {54 vars in 926 bytes} [Sun Jul 24 00:55:18 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:55:18.135192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec67198-7e09-4e3d-aeeb-df80c0ef2788 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:18.135598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec67198-7e09-4e3d-aeeb-df80c0ef2788 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:18.135750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ec67198-7e09-4e3d-aeeb-df80c0ef2788 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:18.202108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3ec67198-7e09-4e3d-aeeb-df80c0ef2788 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:55:18.203357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3ec67198-7e09-4e3d-aeeb-df80c0ef2788 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:18.298902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 743/1492] 10.222.0.18 () {58 vars in 1023 bytes} [Sun Jul 24 00:55:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:55:18.341454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:55:18.342721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:18.445119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f16f2e3-21ce-4ed5-aadc-7802c2daefd6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:55:18.515381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:55:18.516017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:55:18.516297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:55:18.516571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:55:18.518196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:55:18.527510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:18.633801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:55:18.635380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-03bec6f5-5c1c-4cb5-92ed-b2cd4f155b3a service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:55:18.654361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 750/1493] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:55:18 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 314 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:55:28.297065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=82 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:55:28.651057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=50 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:51.840326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:51.840896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:51.840896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:51.912878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:56:51.914070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.007614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 744/1494] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 00:56:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:56:52.016054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.016542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.016646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.091800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:56:52.092933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.186952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 751/1495] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 00:56:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:56:52.191861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.193053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.205134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e336e043-0021-4edd-9083-0e11198fd9e1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:52.266086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:52.299263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:52.299921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.300204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.300487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.301199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 00:56:52.301504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:52.302536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:52.302536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 00:56:52.302536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 00:56:52.303454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:52.314387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 745/1496] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 00:56:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 123 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 00:56:52.322993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.332596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b979b676-be71-4895-bb58-a20745beaa43 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:52.393857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:52.428214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:52.428878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.429156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.429445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.430105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:52.431908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:52.431908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 00:56:52.431908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 00:56:52.433652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:52.487259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 752/1497] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 00:56:52 2022] POST /identity/v3/projects => generated 379 bytes in 165 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:56:52.491644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.493188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.526326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c02cfdf-a640-413b-915a-1846576d52e2 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:52.619670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:52.620907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.621389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.621715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.622410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:52.623435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:52.623435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 00:56:52.623435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 00:56:52.624630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:52.664284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 746/1498] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 00:56:52 2022] POST /identity/v3/users => generated 290 bytes in 173 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 00:56:52.668617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.670141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.734277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88228a73-75c9-445e-8f17-24f4dca5a0f5 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:52.816496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:52.828897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:52.829568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.829851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.830127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.830918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:52.831929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:52.831929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 00:56:52.831929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 00:56:52.832822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:52.845042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 753/1499] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 00:56:52 2022] GET /identity/v3/roles => generated 1420 bytes in 177 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:56:52.849069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.850620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:52.862555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b902a4a-ccee-4573-bd1e-4638b33d3e21 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:52.888214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:52.957582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:52.958253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:52.958538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:52.958844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:52.959594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:52.960605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_roles": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:52.960605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 00:56:52.960605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 00:56:52.961494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c432f35b-71f0-4433-9737-c63c167f1348 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:52.974028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 747/1500] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 00:56:52 2022] GET /identity/v3/roles => generated 1420 bytes in 125 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 00:56:52.978151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:52.980293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.013811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df44f39e-409c-4c05-89c8-9562313419e4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:53.112630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:53.113517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.113800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.114075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] PATH_INFO: `/v3/projects/0907f2ae45b44ad7895a89f54d53acfa/users/b5d7179e1cd44e5d92757fe73c4b87b3/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.136360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] RBAC: Authorizing `identity:create_grant(project_id=0907f2ae45b44ad7895a89f54d53acfa, user_id=b5d7179e1cd44e5d92757fe73c4b87b3, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:53.137043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:53.137043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 00:56:53.137043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 00:56:53.139039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90d8f426-6d40-47aa-9132-692744be508f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:53.162478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 754/1501] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 00:56:52 2022] PUT /identity/v3/projects/0907f2ae45b44ad7895a89f54d53acfa/users/b5d7179e1cd44e5d92757fe73c4b87b3/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 00:56:53.175308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655c532-4a8c-4f93-a507-748f306378ef None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.175600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655c532-4a8c-4f93-a507-748f306378ef None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.175881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3655c532-4a8c-4f93-a507-748f306378ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.231684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3655c532-4a8c-4f93-a507-748f306378ef None None] MFA Rules not processed for user `b5d7179e1cd44e5d92757fe73c4b87b3`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:56:53.232819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3655c532-4a8c-4f93-a507-748f306378ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.309259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 748/1502] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 135 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:56:53.322527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.322835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.323138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.324497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 755/1503] 10.222.0.18 () {54 vars in 964 bytes} [Sun Jul 24 00:56:53 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:56:53.329124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83770de-dcc8-4d26-8e72-ce03bb92eab8 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.329419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83770de-dcc8-4d26-8e72-ce03bb92eab8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.329701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a83770de-dcc8-4d26-8e72-ce03bb92eab8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.360302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:53.403895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a83770de-dcc8-4d26-8e72-ce03bb92eab8 None None] MFA Rules not processed for user `5cba3eae172d459bb5c29b712945691c`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:56:53.405096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a83770de-dcc8-4d26-8e72-ce03bb92eab8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.480251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 749/1504] 10.222.0.18 () {58 vars in 1061 bytes} [Sun Jul 24 00:56:53 2022] POST /identity/v3/auth/tokens => generated 3110 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:56:53.488099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:53.489642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.539299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:53.596242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fb31df9c-4847-4217-954f-4b394982b965 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:53.667309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:53.667956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.668237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.668618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.670280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.773812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:53.775219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:53.779352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:53.794384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 756/1505] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:56:53 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 307 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:56:53.837157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46617cd3-685e-4ee2-93de-eb58713ba10f None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:53.837452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46617cd3-685e-4ee2-93de-eb58713ba10f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:53.837759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46617cd3-685e-4ee2-93de-eb58713ba10f None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:53.839960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 750/1506] 10.222.0.18 () {52 vars in 829 bytes} [Sun Jul 24 00:56:53 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:56:53.843302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:53.844511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:53.942724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f1343d5-fc2f-4a24-8719-2f704e2c36c3 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:54.034339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:54.035163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:54.035441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:54.035711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] PATH_INFO: `/v3/users/b5d7179e1cd44e5d92757fe73c4b87b3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:54.047577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5d7179e1cd44e5d92757fe73c4b87b3)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:54.048601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:54.048601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 00:56:54.048601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 00:56:54.049830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:54.060384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:54.113820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5d7179e1cd44e5d92757fe73c4b87b3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:54.123657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5d7179e1cd44e5d92757fe73c4b87b3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:54.124209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5d7179e1cd44e5d92757fe73c4b87b3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:54.137033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5d7179e1cd44e5d92757fe73c4b87b3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:54.137557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5d7179e1cd44e5d92757fe73c4b87b3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:54.161836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 757/1507] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 00:56:53 2022] DELETE /identity/v3/users/b5d7179e1cd44e5d92757fe73c4b87b3 => generated 0 bytes in 319 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:56:54.166362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:54.166883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:54.167113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:54.222543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:56:54.223770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:54.297632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:54.311593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 751/1508] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 00:56:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 146 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 00:56:54.321350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:54.322595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:54.539802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-148d108d-d446-40ac-a378-a0685f837859 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:54.595374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:56:54.616622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:54.617272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:54.617553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:54.617827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:54.619505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:54.727116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:54.728562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5033381d-8c14-4745-b661-058efe7f228d service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:54.747111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 758/1509] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:56:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 426 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:56:55.067613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:56:55.068982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:56:55.081616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f87c97f-4c7e-4c59-aa8a-e8cb23f5d4c2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:56:55.178164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:56:55.179068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:56:55.179351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:56:55.179625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] PATH_INFO: `/v3/projects/0907f2ae45b44ad7895a89f54d53acfa` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:56:55.191074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0907f2ae45b44ad7895a89f54d53acfa)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:56:55.192103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 00:56:55.192103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 00:56:55.192103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 00:56:55.193324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:56:55.249984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0907f2ae45b44ad7895a89f54d53acfa', 'request_id': 'req-f9135ea5-6e0a-4769-89c8-c089b18b7b39'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:55.255853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0907f2ae45b44ad7895a89f54d53acfa', 'request_id': 'req-f9135ea5-6e0a-4769-89c8-c089b18b7b39'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:55.265834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0907f2ae45b44ad7895a89f54d53acfa', 'request_id': 'req-f9135ea5-6e0a-4769-89c8-c089b18b7b39'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:55.295774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] The token cache is being invalidate because project 0907f2ae45b44ad7895a89f54d53acfa was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 00:56:55.296073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:55.296506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f9135ea5-6e0a-4769-89c8-c089b18b7b39 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 00:56:55.297950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 752/1510] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 00:56:55 2022] DELETE /identity/v3/projects/0907f2ae45b44ad7895a89f54d53acfa => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 00:57:04.744662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=62 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:57:05.293718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=84,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:58:33.915846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:33.916443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:33.916443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:33.917763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 759/1511] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:58:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:33.923418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c787d-a2b8-414c-ad9f-d16b6ac464e8 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:33.923834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c787d-a2b8-414c-ad9f-d16b6ac464e8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:33.923983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d34c787d-a2b8-414c-ad9f-d16b6ac464e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:34.004337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d34c787d-a2b8-414c-ad9f-d16b6ac464e8 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:34.005547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d34c787d-a2b8-414c-ad9f-d16b6ac464e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:34.107787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 753/1512] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:58:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:34.261946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:34.264082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:34.383856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c56c663-c790-41bb-8f2f-078710650b0d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:34.389274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:34.390042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:34.390325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:34.390604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:34.398934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:34.400526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d820567-566b-4b2e-9780-fcbd5ef1fca1 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:34.410461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 760/1513] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:35.236575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd3cc2-0f80-4f7f-ace0-d6876d92c55c None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:35.237149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd3cc2-0f80-4f7f-ace0-d6876d92c55c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:35.237149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4dbd3cc2-0f80-4f7f-ace0-d6876d92c55c None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:35.238523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 754/1514] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:58:35 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:35.243392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b396a87-1a92-4762-886b-adf6e8bddfe0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:35.243841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b396a87-1a92-4762-886b-adf6e8bddfe0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:35.243989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b396a87-1a92-4762-886b-adf6e8bddfe0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:35.261975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1b396a87-1a92-4762-886b-adf6e8bddfe0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:35.263256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1b396a87-1a92-4762-886b-adf6e8bddfe0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:35.272035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 761/1515] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:58:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:35.425523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbd3cc2-0f80-4f7f-ace0-d6876d92c55c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:35.439714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4dbd3cc2-0f80-4f7f-ace0-d6876d92c55c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:35.445349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:35.445983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:35.446270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:35.446546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:35.455254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:35.456842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95ac060f-84eb-4a8d-a077-1d0f92beeba3 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:35.466152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 755/1516] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:36.403061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d315752-db7c-4229-90ad-f0965818984a None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:36.403612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d315752-db7c-4229-90ad-f0965818984a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:36.403612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d315752-db7c-4229-90ad-f0965818984a None None] PATH_INFO: `/v3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:36.404921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 762/1517] 10.222.0.39 () {54 vars in 925 bytes} [Sun Jul 24 00:58:36 2022] GET /identity/v3 => generated 255 bytes in 3 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 00:58:36.411852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa166ce1-3c10-4fd2-9f0f-bfaeedbc8fcb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:36.412276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa166ce1-3c10-4fd2-9f0f-bfaeedbc8fcb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:36.412391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa166ce1-3c10-4fd2-9f0f-bfaeedbc8fcb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:36.468058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-aa166ce1-3c10-4fd2-9f0f-bfaeedbc8fcb None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:36.469839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-aa166ce1-3c10-4fd2-9f0f-bfaeedbc8fcb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:36.485970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:58:36.539238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 756/1518] 10.222.0.39 () {58 vars in 1015 bytes} [Sun Jul 24 00:58:36 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:45.257217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=56 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:58:45.318146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ffe08-7ad3-4652-aa06-6b6498a7054c None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:45.318358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ffe08-7ad3-4652-aa06-6b6498a7054c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:45.318632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d46ffe08-7ad3-4652-aa06-6b6498a7054c None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:45.320036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 763/1519] 10.222.0.18 () {54 vars in 961 bytes} [Sun Jul 24 00:58:45 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:45.324939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3503e-0a50-465c-b541-72b891bd84e7 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:45.325401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3503e-0a50-465c-b541-72b891bd84e7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:45.325531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ed3503e-0a50-465c-b541-72b891bd84e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:45.344242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3ed3503e-0a50-465c-b541-72b891bd84e7 None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:45.346051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3ed3503e-0a50-465c-b541-72b891bd84e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:45.354479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 757/1520] 10.222.0.18 () {58 vars in 1058 bytes} [Sun Jul 24 00:58:45 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:45.362716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46ffe08-7ad3-4652-aa06-6b6498a7054c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:45.376640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d46ffe08-7ad3-4652-aa06-6b6498a7054c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:45.381634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:45.382266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:45.382556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:45.382873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:45.391176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:45.393018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6c73080-a486-4271-9d4c-c8f30ceadb47 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:45.402895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 764/1521] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:45 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:46.710383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f59e8-d686-49ed-807b-5a2dd57c681b None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:46.710794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f59e8-d686-49ed-807b-5a2dd57c681b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:46.711042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1f59e8-d686-49ed-807b-5a2dd57c681b None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:46.712624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 758/1522] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:58:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:46.717541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fa1a8-7e34-41fc-94df-ed7cbfe7cdf0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:46.717971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fa1a8-7e34-41fc-94df-ed7cbfe7cdf0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:46.718088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-028fa1a8-7e34-41fc-94df-ed7cbfe7cdf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:46.737354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-028fa1a8-7e34-41fc-94df-ed7cbfe7cdf0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:46.738554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-028fa1a8-7e34-41fc-94df-ed7cbfe7cdf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:46.746380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 765/1523] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:58:46 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:46.901780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f59e8-d686-49ed-807b-5a2dd57c681b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:46.916883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1f59e8-d686-49ed-807b-5a2dd57c681b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:46.922396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:46.923099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:46.923418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:46.923765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:46.931940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:46.933525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e292f9b9-e545-4910-95ae-a9ac63f4a526 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:46.943363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 759/1524] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 00:58:47.763569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba769728-9507-4ce8-b310-0c275d4e3973 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:47.764150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba769728-9507-4ce8-b310-0c275d4e3973 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:47.764150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba769728-9507-4ce8-b310-0c275d4e3973 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:47.765467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 766/1525] 127.0.0.1 () {54 vars in 901 bytes} [Sun Jul 24 00:58:47 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:47.770390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53e9e6-715e-4658-95d7-df3539d3ccb1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:47.770844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53e9e6-715e-4658-95d7-df3539d3ccb1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:47.771062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b53e9e6-715e-4658-95d7-df3539d3ccb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:47.789966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8b53e9e6-715e-4658-95d7-df3539d3ccb1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:47.791221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8b53e9e6-715e-4658-95d7-df3539d3ccb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:47.800213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 760/1526] 127.0.0.1 () {58 vars in 998 bytes} [Sun Jul 24 00:58:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:47.951167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba769728-9507-4ce8-b310-0c275d4e3973 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:47.965821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba769728-9507-4ce8-b310-0c275d4e3973 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:47.971350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:47.971978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:47.972259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:47.972531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:47.980506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:47.982072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963843c7-9fca-4a62-b986-e90d8af2d18a service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:47.991526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 767/1527] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:51.866720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:51.867312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:51.867312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:51.868610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 761/1528] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:58:51 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:58:51.873656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a61057-ed10-4c0b-89d6-a2dbfa1ab6fc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:51.874051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a61057-ed10-4c0b-89d6-a2dbfa1ab6fc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:51.874197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41a61057-ed10-4c0b-89d6-a2dbfa1ab6fc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:51.893290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-41a61057-ed10-4c0b-89d6-a2dbfa1ab6fc None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:51.894469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-41a61057-ed10-4c0b-89d6-a2dbfa1ab6fc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:51.902162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 768/1529] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:58:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:51.975032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:51.976745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:52.053894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8bc854b7-91de-4537-89b0-08a1eba17be1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:52.057479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:52.058113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:52.058394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:52.058670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:52.068449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:52.069866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6af4916a-b1f5-4952-a678-a343462b4dfa service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:52.079899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 762/1530] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:54.097483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876cc39-7315-4439-82cd-54a81ae49dd2 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:54.098054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876cc39-7315-4439-82cd-54a81ae49dd2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:54.098054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f876cc39-7315-4439-82cd-54a81ae49dd2 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:54.099518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 769/1531] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:58:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:58:54.104735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570966f-ea36-4bfe-b790-d358bfbe000c None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:54.105122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570966f-ea36-4bfe-b790-d358bfbe000c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:54.105281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f570966f-ea36-4bfe-b790-d358bfbe000c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:54.122400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f570966f-ea36-4bfe-b790-d358bfbe000c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:54.123612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f570966f-ea36-4bfe-b790-d358bfbe000c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:54.132060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 763/1532] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:58:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:54.168043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876cc39-7315-4439-82cd-54a81ae49dd2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:54.179306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f876cc39-7315-4439-82cd-54a81ae49dd2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:54.183164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:54.183797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:54.184075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:54.184342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:54.194444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:54.195935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b85e0874-268e-4a90-8b59-65709c204d67 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:54.205532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 770/1533] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:54 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:58:57.763117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:57.763721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:57.763721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:57.765074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 764/1534] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:58:57 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 00:58:57.771114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f97a54-8e20-4089-be0d-3c990e8a3835 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:57.771554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f97a54-8e20-4089-be0d-3c990e8a3835 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:57.771711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43f97a54-8e20-4089-be0d-3c990e8a3835 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:57.789521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-43f97a54-8e20-4089-be0d-3c990e8a3835 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:58:57.790713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-43f97a54-8e20-4089-be0d-3c990e8a3835 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:57.798558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 771/1535] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:58:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:58:58.104928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:58:58.106641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:58:58.171536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:58:58.185298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbcc0bf9-dfa0-4fed-abba-68ba46c130cb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:58:58.188832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:58:58.189468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:58:58.190432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:58:58.190708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:58:58.199733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:58:58.201147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bb0a80d-2576-47b2-83d8-e36efe778bd0 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:58:58.210340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 765/1536] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:58:58 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:59:00.270996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:59:00.271577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:59:00.271577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:59:00.273914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 772/1537] 127.0.0.1 () {54 vars in 916 bytes} [Sun Jul 24 00:59:00 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 00:59:00.280055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621de05-8c9e-437d-a74a-8c9529bb7f16 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:59:00.280582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621de05-8c9e-437d-a74a-8c9529bb7f16 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:59:00.281099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c621de05-8c9e-437d-a74a-8c9529bb7f16 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:59:00.302081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c621de05-8c9e-437d-a74a-8c9529bb7f16 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 00:59:00.303337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c621de05-8c9e-437d-a74a-8c9529bb7f16 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:59:00.311570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 766/1538] 127.0.0.1 () {58 vars in 1013 bytes} [Sun Jul 24 00:59:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 00:59:00.650655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 00:59:00.652388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 00:59:00.669103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7847ef08-e077-4f7f-ae28-335e935f8cbc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 00:59:00.673046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 00:59:00.673694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 00:59:00.674007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 00:59:00.674489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 00:59:00.684477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 00:59:00.685925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3f3f78-3f94-4cb8-838c-59c986ac8aed service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 00:59:00.696866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 773/1539] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 00:59:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 00:59:10.294630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=11 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 00:59:10.692095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=33 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:28.241445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.255685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.255685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.280034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.281274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.292678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 767/1540] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 53 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.301591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173ffd8-bb39-4247-9bf5-836ae624bc83 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.303624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173ffd8-bb39-4247-9bf5-836ae624bc83 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.303624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3173ffd8-bb39-4247-9bf5-836ae624bc83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.332283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3173ffd8-bb39-4247-9bf5-836ae624bc83 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.333556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3173ffd8-bb39-4247-9bf5-836ae624bc83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.345432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 774/1541] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 45 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.351289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.361128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.361473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.361691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.372321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9c619bd-c96d-4bc6-a9a3-07d331e24e2e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.387573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.388557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.389017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.389476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.390514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:28.391020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.392737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.393091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ea52fca-29c4-4812-9bfc-5f0ea8ad6c99 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.393979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.414694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 768/1542] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:28.418757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 775/1543] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 58 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.431991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.432744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e10ea7-0a12-4235-add4-71c893762143 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.433212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e10ea7-0a12-4235-add4-71c893762143 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.433662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91e10ea7-0a12-4235-add4-71c893762143 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.453823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfa2a72-38b8-4bd8-8fdf-70be76ec9435 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.468549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.469732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-91e10ea7-0a12-4235-add4-71c893762143 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.470587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.475117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.476628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-91e10ea7-0a12-4235-add4-71c893762143 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.477096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.477983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.480112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.497909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 769/1544] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 66 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.503838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.504401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.504849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.543826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.546124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.572090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 776/1545] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/projects => generated 375 bytes in 141 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:00:28.573261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.578835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 770/1546] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.581081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.607667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66b719dc-e936-4887-85f4-3f1cf99f9d0f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.713854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fcb29d1-8eac-4808-9942-b7fbca49563d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.723505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.724789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.725069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.725340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.726075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:28.726381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.727432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_domains": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:28.727432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 01:00:28.727432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 01:00:28.728337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63dc3d2b-31b4-4281-a353-c802a1220a83 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.744121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.744932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.745229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.745508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.746200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.748221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fb6e2ed-353c-44f5-b0a9-75c8513bc568 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.758874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 777/1547] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 179 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:28.760478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54718-8226-4f3f-b6b0-0ff5fd91d3a9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.761254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54718-8226-4f3f-b6b0-0ff5fd91d3a9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.761695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36a54718-8226-4f3f-b6b0-0ff5fd91d3a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.783289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-36a54718-8226-4f3f-b6b0-0ff5fd91d3a9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.784820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-36a54718-8226-4f3f-b6b0-0ff5fd91d3a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.790764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 771/1548] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/users => generated 288 bytes in 210 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:28.793304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.793835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.794481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.804010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 778/1549] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.806357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ec0ba-ec1c-4f97-be17-6ff54ee5fcd1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.807111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ec0ba-ec1c-4f97-be17-6ff54ee5fcd1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.807860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f7ec0ba-ec1c-4f97-be17-6ff54ee5fcd1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.817109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.819308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.827488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 772/1550] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.828279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.831825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9f7ec0ba-ec1c-4f97-be17-6ff54ee5fcd1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.833581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9f7ec0ba-ec1c-4f97-be17-6ff54ee5fcd1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.838889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-827ab916-812f-43f0-9d75-20a6b21c7807 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.842134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 779/1551] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.843401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77ffab-d4e6-4762-a186-482ea3209daa None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.843644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.843782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77ffab-d4e6-4762-a186-482ea3209daa None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.843964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef77ffab-d4e6-4762-a186-482ea3209daa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.844309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.844600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.844882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.845577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.846585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:28.846585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 01:00:28.846585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 01:00:28.847829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.865399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ef77ffab-d4e6-4762-a186-482ea3209daa None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.866642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ef77ffab-d4e6-4762-a186-482ea3209daa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.882328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 773/1552] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/projects => generated 377 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:28.882563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 780/1553] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.882690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.882821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.882821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.882821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.900157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.901354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.945763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-830dea4f-6b74-4dc0-8b85-99533dd09d47 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.950676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.951396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.951772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.951947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.952698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.954287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-574b07cf-2dc1-440d-afb8-9fcdc467f550 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.968149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 781/1554] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.968149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:28.968470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 774/1555] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:28 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:28.968470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecb3915-4f52-4802-9673-d56226b6f55b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.968470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecb3915-4f52-4802-9673-d56226b6f55b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.968470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ecb3915-4f52-4802-9673-d56226b6f55b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.974202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-320f86b2-fe42-4f87-bf8b-b2c4adbe5fc3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:28.978919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:28.979609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.979898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.980173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.980894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:28.981205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:28.982879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34c29860-6986-402d-88ff-60c0c9a30b82 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:28.986301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8ecb3915-4f52-4802-9673-d56226b6f55b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:28.987521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8ecb3915-4f52-4802-9673-d56226b6f55b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:28.994792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 782/1556] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:28.996194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 775/1557] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:28.996325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c253c-281e-48f8-8ccb-6ad97ca0a03d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.997364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:28.997652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.997928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:28.999262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c253c-281e-48f8-8ccb-6ad97ca0a03d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:28.999936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d04c253c-281e-48f8-8ccb-6ad97ca0a03d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.016137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.017315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.020326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d04c253c-281e-48f8-8ccb-6ad97ca0a03d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.022119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d04c253c-281e-48f8-8ccb-6ad97ca0a03d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.026697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 776/1558] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.027500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.033656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 783/1559] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.033656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0a88f-f64a-49e6-b43b-763f237528b0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.033656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0a88f-f64a-49e6-b43b-763f237528b0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.033656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2f0a88f-f64a-49e6-b43b-763f237528b0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.037506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-970f9efe-82d9-443b-b2fc-5b65df0c4111 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.042378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.043078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.043367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.043642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.044468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.046394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.050524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a2f0a88f-f64a-49e6-b43b-763f237528b0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.051783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a2f0a88f-f64a-49e6-b43b-763f237528b0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.055621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.060438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 784/1560] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.061747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.062072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.062294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.070905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.073436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 777/1561] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 289 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.074152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.082633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.085102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.087245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5df96ee-3e52-42dc-bd16-ad99b0afad4e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.091766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.092419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.092705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.092983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.093708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.093851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 785/1562] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 33 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.094465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.096040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6efb70-1d91-4476-bbdc-e5ef1139502d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.103890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07a7a06d-68c6-4b31-8531-3b5f4d6b4273 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.109989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 778/1563] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:29.109989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.109989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.109989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.112049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.112725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.113010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.113289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.113973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.116301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.126666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.127927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.136634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 779/1564] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.137424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.142082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 786/1565] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 383 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.143132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.149732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1981211-c6e1-40a2-846b-947d95fc6055 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.206261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-46c5fa4a-dd0b-4bc7-a77a-da029a8db4fc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.209698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.210313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.210604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.210897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.211681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:29.211967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.212353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.213059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.213494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.213645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.213756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.214464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:29.214962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.216630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.226856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 780/1566] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:29.227869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.228553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 787/1567] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:29.229352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.241460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22613864-8698-4172-b17e-5d5bf45a3275 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.243838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9575faa-ca0e-4280-9f60-915cb183f91c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.246089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.246740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.247057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.247337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.248055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:29.248366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.248468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.249116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.249452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.249680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.249956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.250398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.252065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.263586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 781/1568] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:29.264399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.264536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 788/1569] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:29.265329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.279051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d86b7ce0-0933-4c62-ba5c-a6129b777359 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.279423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-def2a185-ccde-49db-a235-1d27875f65c8 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.283759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.283948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.284407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.284691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.284794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.284970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.285106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.285347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] PATH_INFO: `/v3/projects/644464d3831e468f8ab9957bb33817b0/users/a59f5944a0db4710a43a6447019864d2/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.285684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:29.285991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.287954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.299416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 782/1570] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:29.300226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.300734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=644464d3831e468f8ab9957bb33817b0, user_id=a59f5944a0db4710a43a6447019864d2, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.304623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.310593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2e47c89-9895-426a-8919-1255b4f92d8a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.316168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.316902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.317267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.318287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.318948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 789/1571] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:29 2022] PUT /identity/v3/projects/644464d3831e468f8ab9957bb33817b0/users/a59f5944a0db4710a43a6447019864d2/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 54 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:29.319461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:29.319725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.319882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.321722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.334675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 783/1572] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:29.335675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.345096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17918e25-cae5-45f8-bef2-88794431a6f9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.388102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2093288-375f-45ef-b7d0-2116fa3ec9c3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.392859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.393537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.393825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.394120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.394826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.396085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:29.396085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 01:00:29.396085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 01:00:29.397321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.401320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.401977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.402257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.402532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.403256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.405175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.418908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 790/1573] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 292 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.419860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.431026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c32dc8ac-e0ea-422d-b741-61f1300ae7d9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.436227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 784/1574] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 389 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.436227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.452661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.453316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.494307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.494993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.495278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.495548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.496222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.498120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.499557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88dc137b-97d3-46d5-9769-9c5036476a8b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.504610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.505846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.506130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.506846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.507542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.509862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.524562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 791/1575] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 391 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.525765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.536921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 785/1576] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 385 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.537737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.587953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d83cf895-c91d-4d2b-8efd-9f7282772486 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.603168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95fff17-87e1-474f-bc95-f00439d4287b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.607913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.608568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.608848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.609175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.609861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.611862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-068a2610-238f-451a-88ff-8da802f6810e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.639276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 786/1577] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 391 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.640540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.640829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.641106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.647901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.648577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.648864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.649145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.649871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.651545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.659299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:29.660570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:29.662431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 792/1578] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:29.666148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.676045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a979587a-f2b6-465d-86e9-1cba2b080889 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.720976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 787/1579] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:29.723551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.724253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.734080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f82d7d15-5356-41cf-b45f-f9e4a79a6648 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.735134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.735804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.736090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.736364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.737066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.738979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.739256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.739850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.740145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.740419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.741155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.743021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.755497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 788/1580] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:29.756386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.766789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8513b22-35b4-4917-89fb-33a282ce2880 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.767650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 793/1581] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/projects => generated 363 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.767650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.779949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:29.827899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.828669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.828958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.829355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.830044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.832000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.832846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae572d1-2f37-43eb-954e-e23e7f7731af admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.838044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.838715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.839036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.839316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.839996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.841928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.852528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 789/1582] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 295 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.853270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.863521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8afaa3fc-cdfa-4ce4-92af-93ee9a57434e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.868214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 794/1583] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 296 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.868214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.868487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.868952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.869240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.869521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.870203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.872182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.878303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edb57d48-376a-473d-a448-85baf9c462cf admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.884951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.885672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.885960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.886270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.886983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.888906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.892737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 790/1584] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 293 bytes in 40 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.893544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.903740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c27dbb54-11ac-4f32-9b03-f02ce320366e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:29.908374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:29.909246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:29.909535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:29.909810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] PATH_INFO: `/v3/projects/c129535452294d998fee4c9e259c9d41/users/b88e33e967ed47e780b3a1d64c5a0b8a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:29.913918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 795/1585] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:29 2022] POST /identity/v3/users => generated 296 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:29.913918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.925276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] RBAC: Authorizing `identity:create_grant(project_id=c129535452294d998fee4c9e259c9d41, user_id=b88e33e967ed47e780b3a1d64c5a0b8a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:29.926314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:create_grant": "(role:admin and system_scope:all) or ((role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.user.domain_id)s and domain_id:%(target.domain.id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.project.domain_id)s) or (role:admin and domain_id:%(target.group.domain_id)s and domain_id:%(target.domain.id)s)) and (domain_id:%(target.role.domain_id)s or None:%(target.role.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:29.926314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 01:00:29.926314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 01:00:29.928641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:29.943892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 791/1586] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:29 2022] PUT /identity/v3/projects/c129535452294d998fee4c9e259c9d41/users/b88e33e967ed47e780b3a1d64c5a0b8a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:29.944826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:29.969673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b3d33ed-cdb7-4083-90c9-f0d1674d74f7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.010221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c119ea10-dadf-497e-a22e-d9d6478e906e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.014975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.015648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.015948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.016230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.016960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.018694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.020836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.021685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.021967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.022242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.029359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 792/1587] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/roles => generated 1420 bytes in 85 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.030368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.032478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.034176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.040421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61cab4d0-8c04-4195-af3d-7e463bbdea8a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.044066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 796/1588] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:30.044828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.045116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.045836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.046189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.046404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.047127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.049060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.054999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c0c7d72-2f51-4cb7-87bd-9fdc29f17668 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.060017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.060765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.061072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.061382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.062278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.064093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.067998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:30.077224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 793/1589] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:30 2022] POST /identity/v3/users => generated 282 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:30.078102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.083951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 797/1590] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.083951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.088325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20197109-e4f3-44a7-b894-4934e2fdc6af admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.093753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.094414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.094699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.095009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.095274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f9f14c9-feca-413d-a1c3-b8fcdbf9b4c7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.095729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.097339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.101440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.102338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.103246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.103608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.104565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.106488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2830906c-1e24-417f-89c3-0caae2ede334 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.108592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 794/1591] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.109245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.119568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 798/1592] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.120805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-49ed48a6-8eb7-4da9-b124-31ba51bd6103 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.120949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.121123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.121404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.128731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.129664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.130692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.131300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.132208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.135638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.140689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:30.141900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:30.148653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 795/1593] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.149822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.152128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 799/1594] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:30.152557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.163292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-beab1c4f-d720-4b30-8497-50a64786cc53 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.166922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b44e6dab-7a08-4ddd-8326-80b5b0cbfbbb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.169146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.169875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.170162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.170440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.171205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.172128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.172888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.173235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.173593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.173966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] PATH_INFO: `/v3/projects/32d3a8e409644bc0a43293956ccbc83e/users/e6de928916e84bc28fa0204309f17575/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.184429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 800/1595] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.185298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.194820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] RBAC: Authorizing `identity:create_grant(project_id=32d3a8e409644bc0a43293956ccbc83e, user_id=e6de928916e84bc28fa0204309f17575, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.197268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86648ece-b06a-4081-8571-79964087ccbf admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.197746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.201705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:INSERT=11 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:30.202585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.203253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.203541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.203847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.204538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.206174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.217136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 796/1596] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/32d3a8e409644bc0a43293956ccbc83e/users/e6de928916e84bc28fa0204309f17575/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 67 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.217756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 801/1597] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.218466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.221575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.240598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1f8f2c9-60dc-4e4d-86f1-dee8b69e417b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.311014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.312177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.312295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b2a43e4-58a3-428b-ab9e-407020dc2adc admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.312686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.312686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.313009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.314631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.318401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.319364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.319998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.319998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.320625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.322273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.325915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 802/1598] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 108 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.326705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.338576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7f3b181-a01a-4783-a1dd-3ca36bdfb927 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.339037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 797/1599] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 121 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.339929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.341609 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:30.343392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.344055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.344340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.344630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.345334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.347088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.358816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 803/1600] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.360371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.373016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ce146ad-29fd-49c0-8ebc-6a49200a9791 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.377939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.378612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.378935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.379216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.380034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.381730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.395573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 804/1601] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:30.395573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.408720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b146397f-3f7a-4393-bee6-7bf51a6af42f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.421096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a7c4023-ad86-403a-9dbc-b2510712687b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.421869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.423239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.423736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.424362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] PATH_INFO: `/v3/projects/75293bf0100f435bb567b690edc85533/users/9a9256b6d7e24c54b061673cd17217cd/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.425073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.425748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.425999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.426322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.440961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.442750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd801e85-d8fc-4cfe-bde6-986889e4f73f service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.444686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=75293bf0100f435bb567b690edc85533, user_id=9a9256b6d7e24c54b061673cd17217cd, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.447766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.455478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 798/1602] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:30 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:30.457204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.457588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.457971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.484782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 805/1603] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/75293bf0100f435bb567b690edc85533/users/9a9256b6d7e24c54b061673cd17217cd/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.486744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.488564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] MFA Rules not processed for user `e6de928916e84bc28fa0204309f17575`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:30.490502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:30.579843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32349b83-18e3-48c8-9ae2-4cbfc501ce37 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.587202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.588488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.588993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.589467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] PATH_INFO: `/v3/projects/3c68e39bc1724fda980be3463bbf7d7b/users/8f471531dac249d994f864102f7b9d3f/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.602747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 799/1604] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 147 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:30.611644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.618037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:30.631101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab8d5d5-4285-4e36-9449-4565b5ea63d1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.632559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c68e39bc1724fda980be3463bbf7d7b, user_id=8f471531dac249d994f864102f7b9d3f, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.640080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.640422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.641342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.641740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.642171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] PATH_INFO: `/v3/projects/f7d219e82efe44f9b6f860a5fabc7733/users/283a1bfe0e3b45aaa0a33e5fe6b1d85a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.666579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 806/1605] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/3c68e39bc1724fda980be3463bbf7d7b/users/8f471531dac249d994f864102f7b9d3f/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.667602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.675456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f7d219e82efe44f9b6f860a5fabc7733, user_id=283a1bfe0e3b45aaa0a33e5fe6b1d85a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.680203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.695153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:30.705805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 800/1606] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/f7d219e82efe44f9b6f860a5fabc7733/users/283a1bfe0e3b45aaa0a33e5fe6b1d85a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 99 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.711469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:30.764790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2333bf73-0818-460f-aa62-a3280f2d8e0a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.793337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c50a637d-f136-4a95-a1ed-f3cdada599c8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:30.799355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.800700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.801079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.801264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] PATH_INFO: `/v3/projects/a3bfb2d4cca84a1f93fd84ccf4476fee/users/1ad93fa99e8c4e2bb9baa62c6a1e82b8/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.820273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a3bfb2d4cca84a1f93fd84ccf4476fee, user_id=1ad93fa99e8c4e2bb9baa62c6a1e82b8, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.832848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af27ef9-e97d-45bf-affd-081217a20093 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.864405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 801/1607] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/a3bfb2d4cca84a1f93fd84ccf4476fee/users/1ad93fa99e8c4e2bb9baa62c6a1e82b8/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.866397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194de5a9-06e8-481b-a1be-85c14ebdedf5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.866872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194de5a9-06e8-481b-a1be-85c14ebdedf5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.868317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-194de5a9-06e8-481b-a1be-85c14ebdedf5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.883755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:30.886103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.894133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.894494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] PATH_INFO: `/v3/projects/1d5ec73672c04d069684d0a0e0460aa5/users/40b864b9e3c5412fb6b6aa68265d4679/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.911881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-194de5a9-06e8-481b-a1be-85c14ebdedf5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:30.914447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-194de5a9-06e8-481b-a1be-85c14ebdedf5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:30.920272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=1d5ec73672c04d069684d0a0e0460aa5, user_id=40b864b9e3c5412fb6b6aa68265d4679, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:30.926354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0eea1b4d-a637-4c15-bb30-5905bdd1e2ac admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:30.954315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 807/1608] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:30 2022] PUT /identity/v3/projects/1d5ec73672c04d069684d0a0e0460aa5/users/40b864b9e3c5412fb6b6aa68265d4679/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 285 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:30.956160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177abcc3-df56-42f2-8755-e0a10b584209 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.956548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177abcc3-df56-42f2-8755-e0a10b584209 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.956922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-177abcc3-df56-42f2-8755-e0a10b584209 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:30.958962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 808/1609] 10.222.0.18 () {54 vars in 962 bytes} [Sun Jul 24 01:00:30 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:30.960654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bd068-e85c-4206-80ef-d18096e75599 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:30.961043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bd068-e85c-4206-80ef-d18096e75599 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:30.962636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-055bd068-e85c-4206-80ef-d18096e75599 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.031048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 802/1610] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.033554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefaf7a6-1969-4f67-af82-c21f18a3d133 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.034096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefaf7a6-1969-4f67-af82-c21f18a3d133 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.034382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aefaf7a6-1969-4f67-af82-c21f18a3d133 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.035741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-055bd068-e85c-4206-80ef-d18096e75599 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:31.038071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-055bd068-e85c-4206-80ef-d18096e75599 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:31.077607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-aefaf7a6-1969-4f67-af82-c21f18a3d133 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:31.078993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-aefaf7a6-1969-4f67-af82-c21f18a3d133 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:31.124247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 809/1611] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:30 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 165 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.125574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534eade-ea45-4d56-afb3-859b119b6374 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.125872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534eade-ea45-4d56-afb3-859b119b6374 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.126146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c534eade-ea45-4d56-afb3-859b119b6374 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.159826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c534eade-ea45-4d56-afb3-859b119b6374 None None] MFA Rules not processed for user `1ad93fa99e8c4e2bb9baa62c6a1e82b8`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:31.161754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c534eade-ea45-4d56-afb3-859b119b6374 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:31.172804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:31.195471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 803/1612] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.198205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.198920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.199531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.245652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] MFA Rules not processed for user `40b864b9e3c5412fb6b6aa68265d4679`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:31.248183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:31.250551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 810/1613] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 126 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.252006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.252468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.252578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.270065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:31.306599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:31.308813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:31.363128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 804/1614] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 166 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.364295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.401907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 811/1615] 10.222.0.18 () {58 vars in 1059 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 151 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:31.403707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.444523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d504314-586f-44f0-af49-4fa7427eb381 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.449768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.450530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.450944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.451558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.465576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.471310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.486510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 805/1616] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:31.487774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.497462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8695ea1-8ab5-45ae-9a47-5f3e97aca170 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.498433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35445209-ff75-4401-bd6a-32c37c1522f5 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.503792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.504210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.504675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.505211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.505532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.507129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.507966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.508658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.532260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.534137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.545137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.548797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.550756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 806/1617] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 63 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:31.552635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.569163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78eb1b2-66e0-4837-bbb9-4dc83faa138e service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.574336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.575187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.575482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.575761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.576459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.578612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.580101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 812/1618] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:31.582645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.608917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29363ecf-0ef0-43d9-877a-bbb8bfaad8a9 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.621194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.621873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.622154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.624830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.625604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.630535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.631764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 807/1619] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/projects => generated 363 bytes in 80 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:31.632798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.682316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 813/1620] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/projects => generated 391 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:31.684474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.702976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:31.797794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56e8a0db-15a8-42e3-847c-76f948c4e7c6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.804461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.806407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aaee077-0027-4ee0-aeb0-2f8c8a7dab16 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.806542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.806877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.807130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.807959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.811303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.854322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:31.874764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 814/1621] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/users => generated 288 bytes in 192 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:31.875679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.901802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1124a354-067d-4f08-9a00-345aca80ea76 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:31.908037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.908842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.911596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.912202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.914104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:31.916444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:31.964754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 815/1622] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:31 2022] POST /identity/v3/users => generated 302 bytes in 90 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:31.969169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:31.984352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:31.985424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:31.985883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:31.986796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:31.994576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d3372251-fb81-44b3-9003-3708384324eb admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.006596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.007487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.007894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.008276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.009109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.011339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.025714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 816/1623] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:31 2022] GET /identity/v3/roles => generated 1420 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:32.031530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.056087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-361796a6-cf0f-4969-991c-be2f5f7a15c3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.075520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.077304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.077824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.078463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.079734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.090168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.109792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.115704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.130367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 817/1624] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:32 2022] GET /identity/v3/roles => generated 1420 bytes in 102 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:32.130367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.241704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 808/1625] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:31 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 536 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:32.241704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.243875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48ae317a-a840-418f-9e28-c4262a40581f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.243875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.243875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.243875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.244386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] PATH_INFO: `/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab/users/465f5241f20b4ffeba380673dba21f5d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.244465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7926bf9b-0c65-4dac-9c06-5ab1e4e969fd service nova] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.244465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.244465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.244465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.245081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] PATH_INFO: `/v3/projects/d883355ad9f94f6f9b47cab197c0a27f/users/21267bebafee4554838946791e5bc8fd/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.245164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd42beb77d904fdd8f0296aba8b6e5ab, user_id=465f5241f20b4ffeba380673dba21f5d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.245164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.267362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d883355ad9f94f6f9b47cab197c0a27f, user_id=21267bebafee4554838946791e5bc8fd, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.274441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 818/1626] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:32 2022] PUT /identity/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab/users/465f5241f20b4ffeba380673dba21f5d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 143 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:32.274651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.275735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.312029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 809/1627] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:32 2022] PUT /identity/v3/projects/d883355ad9f94f6f9b47cab197c0a27f/users/21267bebafee4554838946791e5bc8fd/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:32.314354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.446760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cbccd3e4-b774-4da6-8df8-ec7d88106b82 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.503744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855d1efd-f684-429a-93ba-f1911e6ce6f8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.526008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.529873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.530196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.530485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.544467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.545991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.588676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 810/1628] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:32 2022] GET /identity/v3/roles => generated 1420 bytes in 275 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:32.592745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.609330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:32.612765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-12b6303d-a1ae-4477-8959-169120fce8dd admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.626366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.628071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.628367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.628683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] PATH_INFO: `/v3/projects/d883355ad9f94f6f9b47cab197c0a27f/users/21267bebafee4554838946791e5bc8fd/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.631940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] RBAC: Authorizing `identity:create_grant(project_id=d883355ad9f94f6f9b47cab197c0a27f, user_id=21267bebafee4554838946791e5bc8fd, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.638931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.650884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.651572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.651935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.652237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.652950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.667801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c5dd5ec-814b-4816-8104-1b164a6333d1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:32.686668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 811/1629] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:32 2022] PUT /identity/v3/projects/d883355ad9f94f6f9b47cab197c0a27f/users/21267bebafee4554838946791e5bc8fd/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 94 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:00:32.692948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 819/1630] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:32 2022] GET /identity/v3/roles => generated 1420 bytes in 418 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:32.702437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:32.743518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e91d5d-03ce-46e2-9b57-6398c73bba28 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.749042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e91d5d-03ce-46e2-9b57-6398c73bba28 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.749677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75e91d5d-03ce-46e2-9b57-6398c73bba28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.801109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:32.886503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-75e91d5d-03ce-46e2-9b57-6398c73bba28 None None] MFA Rules not processed for user `21267bebafee4554838946791e5bc8fd`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:32.887936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-75e91d5d-03ce-46e2-9b57-6398c73bba28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:32.955272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864e7e5d-6f30-4ecd-bc90-9726e1dba419 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:32.969250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:32.970549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:32.971114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:32.974790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] PATH_INFO: `/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab/users/465f5241f20b4ffeba380673dba21f5d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:32.977250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cd42beb77d904fdd8f0296aba8b6e5ab, user_id=465f5241f20b4ffeba380673dba21f5d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:32.993005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4d19154-0e64-405a-b452-408357e5cb08 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:33.030891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 812/1631] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:32 2022] PUT /identity/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab/users/465f5241f20b4ffeba380673dba21f5d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 330 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:33.032576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5b5ce3-1c28-4a94-b8a2-ff23823d4522 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.032876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5b5ce3-1c28-4a94-b8a2-ff23823d4522 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.033158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c5b5ce3-1c28-4a94-b8a2-ff23823d4522 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.034517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 813/1632] 10.222.0.18 () {54 vars in 912 bytes} [Sun Jul 24 01:00:33 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:33.048357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.048810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.049098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.091291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] MFA Rules not processed for user `465f5241f20b4ffeba380673dba21f5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:33.092917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:33.101960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 820/1633] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:32 2022] POST /identity/v3/auth/tokens => generated 3325 bytes in 394 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:33.101960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.101960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.101960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.131242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:33.132700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:33.191423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 814/1634] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:33 2022] POST /identity/v3/auth/tokens => generated 3297 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:33.192077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:33.310069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 821/1635] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:00:33 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 211 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:33.311192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:33.355163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d76ed156-5491-49f8-9554-b96c6dff003d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:33.365191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:33.365884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.366233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.366512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.452721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2b66624-0b64-4d45-b909-4e63756bc212 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:33.463546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:33.464336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.464728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.465159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.481432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:33.494421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:33.498968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:33.516271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:33.522880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:33.524868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:33.526863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 822/1636] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3297 bytes in 216 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:33.563698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 815/1637] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3325 bytes in 372 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:33.624988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:33.635549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82fc7fdb-d9c4-4170-8ec2-14c4d40c9cd1 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:33.651064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:33.651777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.652131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.653398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.687361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:33.690148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:33.733100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 823/1638] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:33.746881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:33.757848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50514f99-2028-4bc3-9643-ac4dfb8905de service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:33.763993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:33.765032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:33.765361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:33.765714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:33.908824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:33.910708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-093c3cd2-2cfe-4fe5-9918-1ab5a374f429 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:33.938839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 816/1639] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:33 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 192 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:34.608956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:34.625941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78f27850-bef4-41be-aaca-89c5dd27443f service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:34.635672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:34.636496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:34.636881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:34.637254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:34.782904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:34.788028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6965be22-05f2-4e36-9023-801583dab05b service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:34.816583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 824/1640] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:34 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 208 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:35.196408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104db0a8-7728-4167-ac38-34248c48579a None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:35.197828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104db0a8-7728-4167-ac38-34248c48579a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:35.199081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-104db0a8-7728-4167-ac38-34248c48579a None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:35.201009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 817/1641] 10.222.0.18 () {54 vars in 962 bytes} [Sun Jul 24 01:00:35 2022] GET /identity => generated 270 bytes in 6 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:35.211040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac3fde-1237-4341-9d22-a850c3625506 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:35.212017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac3fde-1237-4341-9d22-a850c3625506 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:35.212313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cac3fde-1237-4341-9d22-a850c3625506 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:35.248604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4cac3fde-1237-4341-9d22-a850c3625506 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:35.254332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4cac3fde-1237-4341-9d22-a850c3625506 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:35.278122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 825/1642] 10.222.0.18 () {58 vars in 1059 bytes} [Sun Jul 24 01:00:35 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 68 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:35.288755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc859b8-598c-484a-baf5-8b3c2016891c None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:35.289084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc859b8-598c-484a-baf5-8b3c2016891c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:35.289311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4fc859b8-598c-484a-baf5-8b3c2016891c None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:35.291554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 818/1643] 10.222.0.18 () {54 vars in 912 bytes} [Sun Jul 24 01:00:35 2022] GET /identity => generated 270 bytes in 5 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 01:00:35.297227 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa476bc-09a3-4695-b0bb-130df6cc6a88 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:35.297548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa476bc-09a3-4695-b0bb-130df6cc6a88 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:35.297820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aa476bc-09a3-4695-b0bb-130df6cc6a88 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:35.332124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1aa476bc-09a3-4695-b0bb-130df6cc6a88 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:35.333438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1aa476bc-09a3-4695-b0bb-130df6cc6a88 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:35.367346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 826/1644] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:00:35 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 71 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:35.376368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc859b8-598c-484a-baf5-8b3c2016891c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:35.391761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4fc859b8-598c-484a-baf5-8b3c2016891c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:35.402628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:35.403502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:35.405713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:35.406279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:35.430280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:35.436359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c279835b-2a05-4ea9-8184-75b1bc048aa8 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:35.468985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 819/1645] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:35 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:41.987507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:41.988035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:41.988035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:42.041591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] MFA Rules not processed for user `b88e33e967ed47e780b3a1d64c5a0b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:42.043209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:42.164732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 827/1646] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:41 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 170 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:42.166758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816c012-fd61-4567-92c2-a689e9622238 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:42.167369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816c012-fd61-4567-92c2-a689e9622238 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:42.167369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7816c012-fd61-4567-92c2-a689e9622238 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:42.221813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7816c012-fd61-4567-92c2-a689e9622238 None None] MFA Rules not processed for user `b88e33e967ed47e780b3a1d64c5a0b8a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:42.223100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7816c012-fd61-4567-92c2-a689e9622238 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:42.250269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 820/1647] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:42 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:42.262476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:42.282900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f8836d8a-aa24-4764-b3db-42482759a690 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:42.297606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:42.298525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:42.299187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:42.299637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:42.326971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:42.328892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-910437d3-c47f-4659-8e01-a32877be506f service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:42.350315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 828/1648] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:42 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:43.756753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddfd1fa-715a-4ea2-9198-340d07c10a86 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:43.757260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddfd1fa-715a-4ea2-9198-340d07c10a86 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:43.757349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ddfd1fa-715a-4ea2-9198-340d07c10a86 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:43.758714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 821/1649] 10.222.0.18 () {54 vars in 912 bytes} [Sun Jul 24 01:00:43 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:43.772407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:43.772944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:43.773022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:43.854693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:43.857093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:43.935211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:44.016261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:44.016261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:44.016261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:44.037763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 829/1650] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:00:43 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 266 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:00:44.076981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] MFA Rules not processed for user `a59f5944a0db4710a43a6447019864d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:44.078875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:44.275979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 822/1651] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:44 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 256 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:44.288837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:44.315386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab23961-7707-4a10-8d4c-8367b835653e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:44.333808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:44.334505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:44.334840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:44.335122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:44.365585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:44.367476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:44.415213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 830/1652] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:44 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 127 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:44.417731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:44.430216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6e7a082a-c4d7-424e-b62f-bfdbecd3cf24 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:44.435011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:44.436151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:44.436596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:44.437191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:44.453474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:44.456418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6c0a24c-4ebb-4421-aa18-71c7bbed9926 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:44.467347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 823/1653] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:44 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:45.264162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:45.474346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-872080a5-f13e-4c13-9672-a7b51d22e941 service nova] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:45.497893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:45.498830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:45.499121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:45.499401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] PATH_INFO: `/v3/users/40b864b9e3c5412fb6b6aa68265d4679` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:45.505803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] RBAC: Authorizing `identity:delete_user(user_id=40b864b9e3c5412fb6b6aa68265d4679)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:45.508236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:45.627172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:45.628016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '40b864b9e3c5412fb6b6aa68265d4679'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:45.628207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:45.628631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:45.643243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '40b864b9e3c5412fb6b6aa68265d4679'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:45.655154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '40b864b9e3c5412fb6b6aa68265d4679'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:45.671496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '40b864b9e3c5412fb6b6aa68265d4679'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:45.672055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e2e3210e-30fa-42fc-bca7-db28be0b5cf2 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '40b864b9e3c5412fb6b6aa68265d4679'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:45.704868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] MFA Rules not processed for user `9a9256b6d7e24c54b061673cd17217cd`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:45.706125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:45.744181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 831/1654] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:45 2022] DELETE /identity/v3/users/40b864b9e3c5412fb6b6aa68265d4679 => generated 0 bytes in 480 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:45.748285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:45.748802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:45.749237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:45.816282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:45.835978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:45.837236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:45.954030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 824/1655] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 328 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:45.959622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:45.962125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:45.982351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:46.013827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 832/1656] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:45 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 267 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:46.026400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:46.027885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:46.034873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38df5ca8-9803-4ebc-b959-79252c0140ca None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:46.223881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38987bf9-36b0-4586-a3ab-b89294431322 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:46.230603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:46.231517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.232149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.232567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.237774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:46.238415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.238836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.239261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.276026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:46.282520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45c6e09d-9093-4151-a3be-62b4b5a2418f service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:46.330138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 833/1657] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:46 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 304 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:46.450187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:46.452421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6656adfc-9d53-44da-b693-1373c8991c5c service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:46.474528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 825/1658] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:45 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 515 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:00:46.541812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ee5dc-1d07-441e-bb9c-825c2ff0a192 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.542108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ee5dc-1d07-441e-bb9c-825c2ff0a192 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.542387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-058ee5dc-1d07-441e-bb9c-825c2ff0a192 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.543882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 834/1659] 10.222.0.18 () {54 vars in 961 bytes} [Sun Jul 24 01:00:46 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:46.552203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.552534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.552817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.601098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:46.602361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:46.775225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 826/1660] 10.222.0.18 () {58 vars in 1058 bytes} [Sun Jul 24 01:00:46 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 225 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:46.803468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ee5dc-1d07-441e-bb9c-825c2ff0a192 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:46.836732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-058ee5dc-1d07-441e-bb9c-825c2ff0a192 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:46.853054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:46.855283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:46.856311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:46.859345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.859573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.859859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.865390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:46.866222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:46.880795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6682159-8b27-41cf-ba03-afa7154f58e4 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:46.895854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:46.897493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:46.897828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e86bf81-fe0d-4daa-aaa0-89d3f93a5218 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:46.898399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:46.898721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:46.898990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] PATH_INFO: `/v3/users/e6de928916e84bc28fa0204309f17575` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:46.900219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e6de928916e84bc28fa0204309f17575)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:46.901236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: WARNING py.warnings [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_user": "(role:admin and system_scope:all) or (role:admin and token.domain.id:%(target.user.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:46.901236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: warnings.warn(msg) Jul 24 01:00:46.901236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: Jul 24 01:00:46.902460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:46.920765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 835/1661] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:46 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 128 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:47.070414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e6de928916e84bc28fa0204309f17575'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:47.095808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e6de928916e84bc28fa0204309f17575'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:47.096660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e6de928916e84bc28fa0204309f17575'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:47.124581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e6de928916e84bc28fa0204309f17575'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:47.186171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e6de928916e84bc28fa0204309f17575'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:47.202917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 827/1662] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:46 2022] DELETE /identity/v3/users/e6de928916e84bc28fa0204309f17575 => generated 0 bytes in 348 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:47.204865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:47.205578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:47.205718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:47.251735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:47.266762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:47.550864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 836/1663] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 347 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:00:47.607583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:47.609097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:47.772225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f6b376-f5c5-46ab-85df-d925053aa552 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:47.790501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:47.792593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:47.793072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:47.793796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:47.813466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:47.827247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:47.854737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:47.856422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:47.867450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 828/1664] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 260 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:47.892142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16f3468a-e561-4755-9818-2fa2d872f57e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:47.910206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:47.911492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:47.911846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:47.913346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] PATH_INFO: `/v3/projects/1d5ec73672c04d069684d0a0e0460aa5` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:47.923755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] RBAC: Authorizing `identity:delete_project(project_id=1d5ec73672c04d069684d0a0e0460aa5)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:47.924992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: WARNING py.warnings [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] /usr/local/lib/python3.8/dist-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:delete_project": "(role:admin and system_scope:all) or (role:admin and domain_id:%(target.project.domain_id)s)" failed scope check. The token used to make the request was project scoped but the policy requires ['system', 'domain'] scope. This behavior may change in the future where using the intended scope is required Jul 24 01:00:47.924992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: warnings.warn(msg) Jul 24 01:00:47.924992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: Jul 24 01:00:47.931255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:48.029963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1d5ec73672c04d069684d0a0e0460aa5', 'request_id': 'req-94e6dc76-54c9-4471-86e9-0fdeafc0196a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.042994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1d5ec73672c04d069684d0a0e0460aa5', 'request_id': 'req-94e6dc76-54c9-4471-86e9-0fdeafc0196a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.052975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1d5ec73672c04d069684d0a0e0460aa5', 'request_id': 'req-94e6dc76-54c9-4471-86e9-0fdeafc0196a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.095680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] The token cache is being invalidate because project 1d5ec73672c04d069684d0a0e0460aa5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:00:48.096226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.097061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.101542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 837/1665] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:00:47 2022] DELETE /identity/v3/projects/1d5ec73672c04d069684d0a0e0460aa5 => generated 0 bytes in 247 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:00:48.106037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:48.125551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4fabcc1-94d7-4ec4-b9f7-1d5124653961 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:48.357197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:48.365784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:48.366363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:48.367637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] PATH_INFO: `/v3/users/21267bebafee4554838946791e5bc8fd` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:48.374476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] RBAC: Authorizing `identity:delete_user(user_id=21267bebafee4554838946791e5bc8fd)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:48.378967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:48.423541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:48.488391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '21267bebafee4554838946791e5bc8fd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.510445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '21267bebafee4554838946791e5bc8fd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.514814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '21267bebafee4554838946791e5bc8fd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.523903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '21267bebafee4554838946791e5bc8fd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.551508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c9e49141-e327-4b05-ae1b-488e6c65f58d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '21267bebafee4554838946791e5bc8fd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.565247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 829/1666] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:48 2022] DELETE /identity/v3/users/21267bebafee4554838946791e5bc8fd => generated 0 bytes in 457 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:48.661069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:48.663246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:48.677276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:48.682706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e6dc76-54c9-4471-86e9-0fdeafc0196a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:48.822882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:48.824166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:48.824695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:48.825235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] PATH_INFO: `/v3/projects/32d3a8e409644bc0a43293956ccbc83e` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:48.826831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32d3a8e409644bc0a43293956ccbc83e)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:48.829305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:48.892226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32d3a8e409644bc0a43293956ccbc83e', 'request_id': 'req-912ae313-8618-4773-a34e-45f26e062771'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.904175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32d3a8e409644bc0a43293956ccbc83e', 'request_id': 'req-912ae313-8618-4773-a34e-45f26e062771'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.916766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32d3a8e409644bc0a43293956ccbc83e', 'request_id': 'req-912ae313-8618-4773-a34e-45f26e062771'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.932955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] The token cache is being invalidate because project 32d3a8e409644bc0a43293956ccbc83e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:00:48.933567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.934484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-912ae313-8618-4773-a34e-45f26e062771 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:48.940017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 838/1667] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:00:48 2022] DELETE /identity/v3/projects/32d3a8e409644bc0a43293956ccbc83e => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:48.942752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:48.943350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:48.943953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:48.965789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:48.967549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:49.053715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 830/1668] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 113 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:49.055526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.056581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.057090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.095865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:49.097637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:49.125298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.127396 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 839/1669] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 73 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:49.140478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209cca6a-0027-421f-8335-636f2b7d7a4d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.153363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.154057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.154398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.154615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.155443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:49.155753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.157452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.192851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 831/1670] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:00:49.194393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.210188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a1992c5-6142-4153-8c8f-0082d4fce3e0 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.231763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.232437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.232720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.232997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.233794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.235870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.302514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 840/1671] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:49 2022] POST /identity/v3/projects => generated 379 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:49.305422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.345008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-599d376e-8fc0-4171-b532-ec7ecdd52290 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.595542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.598147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.598519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.605173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.606241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.614106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.671544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 832/1672] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:49 2022] POST /identity/v3/users => generated 290 bytes in 366 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:00:49.673131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.694918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6892a63c-6cc4-4078-ba70-04ee915fba35 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.697680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.700290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:49.708161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.709376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.710684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.711747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.712791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.717669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.729208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168f6774-39e4-468a-8b00-619bd32b478d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.738131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 841/1673] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:49 2022] GET /identity/v3/roles => generated 1420 bytes in 65 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:49.739597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.747269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.748724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.749006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.749295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] PATH_INFO: `/v3/users/1ad93fa99e8c4e2bb9baa62c6a1e82b8` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.753958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] RBAC: Authorizing `identity:delete_user(user_id=1ad93fa99e8c4e2bb9baa62c6a1e82b8)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.759483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.779665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:49.791061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aee4964-4971-4cb9-8962-6e9b06ecfcd3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.803728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.804522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.804810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.805095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.807233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.812669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.833661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 842/1674] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:49 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:49.834853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:49.848921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69e81385-be29-4ebe-95d3-9f339b290871 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:49.856066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:49.857359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.857863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.858296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] PATH_INFO: `/v3/projects/34ab3c8f771f40b6b9c2598ab037ed7e/users/8bd9b8d822c643c1bc7cbfceba65842a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.880423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=34ab3c8f771f40b6b9c2598ab037ed7e, user_id=8bd9b8d822c643c1bc7cbfceba65842a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:49.883901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf4dc20-e9c7-4240-91da-e4dbb5476b69 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:49.912106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1ad93fa99e8c4e2bb9baa62c6a1e82b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:49.913463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 843/1675] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:49 2022] PUT /identity/v3/projects/34ab3c8f771f40b6b9c2598ab037ed7e/users/8bd9b8d822c643c1bc7cbfceba65842a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:49.915590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.916111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.916574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:49.927545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1ad93fa99e8c4e2bb9baa62c6a1e82b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:49.928226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1ad93fa99e8c4e2bb9baa62c6a1e82b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:49.946013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1ad93fa99e8c4e2bb9baa62c6a1e82b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:49.951640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] MFA Rules not processed for user `8bd9b8d822c643c1bc7cbfceba65842a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:49.953205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:49.965331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:49.968545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a4283c64-a6c3-4ec9-bfd4-94eca732e83e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1ad93fa99e8c4e2bb9baa62c6a1e82b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:49.990149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 833/1676] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:49 2022] DELETE /identity/v3/users/1ad93fa99e8c4e2bb9baa62c6a1e82b8 => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:49.991771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:49.995734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:49.996244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.045592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 844/1677] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:49 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:50.050357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.051916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:50.063812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:50.064956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:50.202385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90633bd5-5a28-4aee-83be-1ec599beb81b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.250670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 834/1678] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 260 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:50.277259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:50.278146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.278484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.278784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.279568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:50.282300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:50.332127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 845/1679] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:50 2022] POST /identity/v3/projects => generated 379 bytes in 282 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:50.336886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.351208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23184423-e6f7-435d-a2ee-07a4ba7b6240 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.436802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.443996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:50.522728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:50.523535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.523849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.524502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.525295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:50.527592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:50.566593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 835/1680] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:50 2022] POST /identity/v3/users => generated 296 bytes in 230 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:00:50.571600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.587896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c6e839c8-a182-4107-9624-0a6f700c7ddb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.594184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:50.595405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.595874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.596750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.598731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:50.601481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:50.665820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 836/1681] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:00:50.671726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:50.680734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.700557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=86,keystone:DELETE=10,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:50.712506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0394a7c1-75f9-46cc-ab19-9a994ce3b13a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.713949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b28cdf11-2d58-422f-be87-08b76cde2b6c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.734654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:50.735397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.735726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.735955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.736376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:50.737927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.738298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.738824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] PATH_INFO: `/v3/projects/89c86c44547f4bc5b7d217b380008c74/users/7236f76bc19845ee8cd115c11f1088a4/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:50.772594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:50.775941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:50.787990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89c86c44547f4bc5b7d217b380008c74, user_id=7236f76bc19845ee8cd115c11f1088a4, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:50.794980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c9a526-bc8c-4580-afb0-9ec64a396af0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:50.809043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 846/1682] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:50 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 373 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:50.843404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 837/1683] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:50 2022] PUT /identity/v3/projects/89c86c44547f4bc5b7d217b380008c74/users/7236f76bc19845ee8cd115c11f1088a4/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 176 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:50.850297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:50.873661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1026089a-c35a-4866-8ed3-0c312f2da0a1 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:50.982731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d3fcb-a8f9-4e26-aced-048dae3c747c None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:50.984822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d3fcb-a8f9-4e26-aced-048dae3c747c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:50.985401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-180d3fcb-a8f9-4e26-aced-048dae3c747c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:51.154706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-180d3fcb-a8f9-4e26-aced-048dae3c747c None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:51.158009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-180d3fcb-a8f9-4e26-aced-048dae3c747c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:51.169627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:51.171142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:51.171536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:51.173060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:51.179316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:51.191213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:51.233034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 847/1684] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:50 2022] GET /identity/v3/roles => generated 1420 bytes in 383 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:51.237335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:51.291624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d626ecf6-6891-446e-8b95-cebf41c51614 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:51.308742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:51.310275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:51.311863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:51.312278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] PATH_INFO: `/v3/projects/89c86c44547f4bc5b7d217b380008c74/users/7236f76bc19845ee8cd115c11f1088a4/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:51.320276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] RBAC: Authorizing `identity:create_grant(project_id=89c86c44547f4bc5b7d217b380008c74, user_id=7236f76bc19845ee8cd115c11f1088a4, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:51.348053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:51.398591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 838/1685] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:00:50 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 417 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:51.420518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 848/1686] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:51 2022] PUT /identity/v3/projects/89c86c44547f4bc5b7d217b380008c74/users/7236f76bc19845ee8cd115c11f1088a4/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:51.433059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:51.433353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:51.433631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:51.491810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] MFA Rules not processed for user `7236f76bc19845ee8cd115c11f1088a4`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:51.494913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:51.711955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 839/1687] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:51 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 281 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:51.730990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:51.733028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:51.737516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:51.739638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:51.882380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:51.934101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a539608-877b-400b-ade6-a59066c2f598 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:51.946392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:51.949104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:51.949665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:51.951191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:51.991969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:51.993467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:51.994486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d79058ea-0b29-473f-8793-3f26b7b3b2c0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:52.008059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:52.008494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:52.008769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:52.009110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:52.018501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 849/1688] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:51 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 288 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:52.161654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:52.163494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-815d33e6-7a7b-4387-8232-f17a3bad13e4 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:52.195218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 840/1689] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:00:51 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 458 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 01:00:52.304988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:52.320387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735379bd-acb9-47b9-84f6-44c3cf784669 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:52.333021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:52.334244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:52.334709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:52.335868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:52.528630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:52.533377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c41bd07-91a2-45a4-bd63-1a8f76595a8f service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:52.563952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 850/1690] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:52 2022] GET /identity/v3/auth/tokens => generated 3246 bytes in 259 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:53.194310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4207456-dc18-4e19-a9fe-76b227d65171 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:53.196790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4207456-dc18-4e19-a9fe-76b227d65171 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:53.197292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4207456-dc18-4e19-a9fe-76b227d65171 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:53.249813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b4207456-dc18-4e19-a9fe-76b227d65171 None None] MFA Rules not processed for user `a59f5944a0db4710a43a6447019864d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:53.255035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b4207456-dc18-4e19-a9fe-76b227d65171 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:53.411004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 841/1691] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 218 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:53.921239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:53.922067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:53.922494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:53.954117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:53.972176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] MFA Rules not processed for user `283a1bfe0e3b45aaa0a33e5fe6b1d85a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:53.975531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:54.245608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 851/1692] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:53 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 325 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:54.256740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:54.257632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:54.258051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:54.304558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] MFA Rules not processed for user `283a1bfe0e3b45aaa0a33e5fe6b1d85a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:54.315603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:54.342462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:54.344056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:54.357040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 842/1693] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:54 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:54.369400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:54.371241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:54.572535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:54.637985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeea1dc4-1611-47b8-ad9d-a9c47a45319e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:54.644224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:54.645481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:54.645934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:54.646220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] PATH_INFO: `/v3/projects/d883355ad9f94f6f9b47cab197c0a27f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:54.651735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=d883355ad9f94f6f9b47cab197c0a27f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:54.666460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:54.667523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d292a5da-f0a9-457e-927a-a708f7c91537 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:54.695538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:54.698835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:54.703180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:54.703180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:54.767412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:54.772055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-860ffd1b-438c-4f85-8eb0-f1378cdf5954 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:54.829409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 843/1694] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:54 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 460 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:54.872429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd883355ad9f94f6f9b47cab197c0a27f', 'request_id': 'req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:54.899216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd883355ad9f94f6f9b47cab197c0a27f', 'request_id': 'req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:54.926223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd883355ad9f94f6f9b47cab197c0a27f', 'request_id': 'req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:54.989383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] The token cache is being invalidate because project d883355ad9f94f6f9b47cab197c0a27f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:00:54.990831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:54.992970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c0927f13-e933-4fb9-a5b4-e30fc2ff1eb6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:54.999057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 852/1695] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:00:54 2022] DELETE /identity/v3/projects/d883355ad9f94f6f9b47cab197c0a27f => generated 0 bytes in 657 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:00:55.004606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a788be8b-d93e-404e-9b36-c46b447f4eda None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.006619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a788be8b-d93e-404e-9b36-c46b447f4eda None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.008639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a788be8b-d93e-404e-9b36-c46b447f4eda None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.098925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a788be8b-d93e-404e-9b36-c46b447f4eda None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:55.102637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a788be8b-d93e-404e-9b36-c46b447f4eda None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:55.143433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.143738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.144004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.239241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] MFA Rules not processed for user `9a9256b6d7e24c54b061673cd17217cd`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:55.243181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:55.342912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 844/1696] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 340 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:55.357548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.357840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.358121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.394548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:55.396031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:55.406668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 845/1697] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 50 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:55.407412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:55.409478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:55.413851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:55.450584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66df5bb8-061b-4d84-872b-43736008d83d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:55.463726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:55.466826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.466826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.466826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] PATH_INFO: `/v3/projects/a3bfb2d4cca84a1f93fd84ccf4476fee` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.470953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a3bfb2d4cca84a1f93fd84ccf4476fee)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:55.477682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:55.526233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 853/1698] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:55 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 384 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:55.527235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:55.567598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c34ee8b4-ce0d-469d-9f36-b925a14f2572 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:55.592158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:55.594549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.596170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.596860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.601325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:55.602591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:55.606879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a3bfb2d4cca84a1f93fd84ccf4476fee', 'request_id': 'req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:55.608534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:55.614816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a3bfb2d4cca84a1f93fd84ccf4476fee', 'request_id': 'req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:55.628449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a3bfb2d4cca84a1f93fd84ccf4476fee', 'request_id': 'req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:55.658791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 854/1699] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 132 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:55.659796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:55.664167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] The token cache is being invalidate because project a3bfb2d4cca84a1f93fd84ccf4476fee was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:00:55.664492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:55.666563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:55.670805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 846/1700] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:00:55 2022] DELETE /identity/v3/projects/a3bfb2d4cca84a1f93fd84ccf4476fee => generated 0 bytes in 263 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:55.675097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:55.679480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:55.690490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2538944-78ff-4639-aff2-f35853001ac8 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:55.891221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:55.905239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.906265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.906704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.908155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:55.908700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d535c64-7c21-4bde-ad7c-aa13d62c31e9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:55.911604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:55.929828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:55.930746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:55.931062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:55.931348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] PATH_INFO: `/v3/users/465f5241f20b4ffeba380673dba21f5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:55.934250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] RBAC: Authorizing `identity:delete_user(user_id=465f5241f20b4ffeba380673dba21f5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:55.936261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:55.938291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:56.023698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 855/1701] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:55 2022] POST /identity/v3/projects => generated 389 bytes in 364 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:56.028501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:56.033027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:56.081674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '465f5241f20b4ffeba380673dba21f5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:56.099122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '465f5241f20b4ffeba380673dba21f5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:56.099781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '465f5241f20b4ffeba380673dba21f5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:56.110507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '465f5241f20b4ffeba380673dba21f5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:56.137625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '465f5241f20b4ffeba380673dba21f5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:56.156648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 847/1702] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:55 2022] DELETE /identity/v3/users/465f5241f20b4ffeba380673dba21f5d => generated 0 bytes in 477 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:56.203588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2259df01-0024-4a27-a02b-897e663f0ad7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:56.324852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:56.340973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:56.341828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:56.343764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:56.343764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:56.343947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:56.345901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:56.401887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 856/1703] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:56 2022] POST /identity/v3/users => generated 295 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:56.405755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:56.407402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:56.436715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f07b39-5d83-42b9-b940-258ba8348145 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:56.451900 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:56.452592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:56.452875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:56.453155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:56.453918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:56.455605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:56.482962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 848/1704] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:56 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:56.491596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:56.520296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff77ab91-f744-4242-b512-c69a92e5489b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:56.536904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:56.538943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:56.540672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:56.541754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:56.544261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:56.550192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4eeaa3a6-6105-4f1a-a188-6f242b5525f4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:56.598954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 857/1705] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:56 2022] GET /identity/v3/roles => generated 1420 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:56.603278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:56.618728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ac62d6a-e0ac-4315-b179-ff44d8f134fe admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:56.636164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:56.637686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:56.637984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:56.638267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] PATH_INFO: `/v3/projects/2a44d084328f4197b505ead7127d47a0/users/000af482c7fb44e385f255123f94db53/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:56.711770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] RBAC: Authorizing `identity:create_grant(project_id=2a44d084328f4197b505ead7127d47a0, user_id=000af482c7fb44e385f255123f94db53, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:56.723747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:56.763946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 849/1706] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:56 2022] PUT /identity/v3/projects/2a44d084328f4197b505ead7127d47a0/users/000af482c7fb44e385f255123f94db53/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:56.775471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:56.775828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:56.776116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:56.852635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] MFA Rules not processed for user `000af482c7fb44e385f255123f94db53`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:56.856718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.039591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.044015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.082053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 858/1707] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:56 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 307 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:57.087415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.266810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1c4ff6b9-ebdc-492d-be48-f0522a05d374 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.281929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.282847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.283141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.283469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] PATH_INFO: `/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.284968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cd42beb77d904fdd8f0296aba8b6e5ab)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.289394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.291338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0678bf6c-f305-425b-879a-43d79355b0b7 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.306431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.307191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.307457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.307774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.308479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.313087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75aaf49c-0132-4c11-b000-90f64350ccb6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.369688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 859/1708] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/projects => generated 389 bytes in 283 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:57.389508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e5559-89cf-42bd-9c3d-44b8333ebf48 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.390114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e5559-89cf-42bd-9c3d-44b8333ebf48 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.390684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-046e5559-89cf-42bd-9c3d-44b8333ebf48 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.403450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 860/1709] 10.222.0.18 () {54 vars in 912 bytes} [Sun Jul 24 01:00:57 2022] GET /identity => generated 270 bytes in 33 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:00:57.404657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046e5559-89cf-42bd-9c3d-44b8333ebf48 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.427383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:57.445811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cd42beb77d904fdd8f0296aba8b6e5ab', 'request_id': 'req-a45e19a6-7c0b-42a2-bf50-198e32c2df33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:57.463026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cd42beb77d904fdd8f0296aba8b6e5ab', 'request_id': 'req-a45e19a6-7c0b-42a2-bf50-198e32c2df33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:57.475998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cd42beb77d904fdd8f0296aba8b6e5ab', 'request_id': 'req-a45e19a6-7c0b-42a2-bf50-198e32c2df33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:57.500987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] The token cache is being invalidate because project cd42beb77d904fdd8f0296aba8b6e5ab was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:00:57.501768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:57.502517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a45e19a6-7c0b-42a2-bf50-198e32c2df33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:57.508620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:57.518325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 850/1710] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:00:57 2022] DELETE /identity/v3/projects/cd42beb77d904fdd8f0296aba8b6e5ab => generated 0 bytes in 471 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:57.524010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618d8315-8869-45e6-a99d-7b4d33542c87 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.524318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618d8315-8869-45e6-a99d-7b4d33542c87 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.524582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618d8315-8869-45e6-a99d-7b4d33542c87 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.537647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-046e5559-89cf-42bd-9c3d-44b8333ebf48 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.543787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.544859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.545373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.545900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.547055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.550736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52a6c81b-1d2f-4edb-a810-dd19de1f88d0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.583216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-618d8315-8869-45e6-a99d-7b4d33542c87 None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:57.584696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-618d8315-8869-45e6-a99d-7b4d33542c87 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.601101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 861/1711] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/users => generated 301 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:57.603315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fb130-294a-404f-b959-0180fada0196 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.604684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fb130-294a-404f-b959-0180fada0196 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.605007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-669fb130-294a-404f-b959-0180fada0196 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.631292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-669fb130-294a-404f-b959-0180fada0196 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:57.632955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-669fb130-294a-404f-b959-0180fada0196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.658488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 862/1712] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:57.658488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669fb130-294a-404f-b959-0180fada0196 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.658488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-669fb130-294a-404f-b959-0180fada0196 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.701855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-669fb130-294a-404f-b959-0180fada0196 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.719501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 851/1713] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:57.721513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.721998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.722551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.749345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:57.750620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.751743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.751986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.752303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.753024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.753235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.755019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.771596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 852/1714] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 51 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:57.772976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.784388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57805e56-7633-4d8a-aa4a-a9cffb278e37 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.790872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 863/1715] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:57 2022] GET /identity/v3/roles => generated 1420 bytes in 139 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:57.796639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.797059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.798589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.799069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.799823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.800705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:00:57.801293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.804214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.820074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a598f19-b250-4c77-a3e2-a83f21f0a33b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.827265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 853/1716] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:00:57 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:00:57.844992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.846951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.846951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.846951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.846951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] PATH_INFO: `/v3/projects/82fc5a79fc054d4ca04ff73f39b40889/users/69eb16f5d6bd4815af02f5471809d2c7/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.856279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ce6047b-b12e-4cd6-a0dc-46431b5b8a8f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:57.887321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:57.887550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82fc5a79fc054d4ca04ff73f39b40889, user_id=69eb16f5d6bd4815af02f5471809d2c7, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.889131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:57.890016 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:57.890505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.891031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:57.893208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:57.900586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:57.932621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 864/1717] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:57 2022] PUT /identity/v3/projects/82fc5a79fc054d4ca04ff73f39b40889/users/69eb16f5d6bd4815af02f5471809d2c7/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:57.936167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:57.948434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:57.968562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 854/1718] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:57 2022] POST /identity/v3/projects => generated 387 bytes in 140 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:57.969513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.154388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-63642fc9-bcdc-4c7d-b3fc-87887315e354 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.206861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0489caf-83ee-4371-b401-35abd5505481 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.215510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.216579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.217123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.217621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.218570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.221293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.253682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 855/1719] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:57 2022] GET /identity/v3/roles => generated 1420 bytes in 284 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:58.257014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.305444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.306121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.306412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.306699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.327326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bff298f0-967d-44cc-8d0c-71fc9f1d88f9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.327716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.329271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.356890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 865/1720] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:57 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 423 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:58.358420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.385447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.388119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73701334-d3f2-4dfa-8f5e-28f61a87b184 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.388570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.389020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.389971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.390816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.397465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.431291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.432278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.432577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.432847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] PATH_INFO: `/v3/projects/82fc5a79fc054d4ca04ff73f39b40889/users/69eb16f5d6bd4815af02f5471809d2c7/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.446601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] RBAC: Authorizing `identity:create_grant(project_id=82fc5a79fc054d4ca04ff73f39b40889, user_id=69eb16f5d6bd4815af02f5471809d2c7, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.449361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba74adc3-1f5f-49aa-9b2b-a7ea6be46c27 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.451305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 856/1721] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:58 2022] POST /identity/v3/users => generated 294 bytes in 197 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:58.460988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.475205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef7b563e-2daa-425c-9e1b-a7574023436c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.499813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 866/1722] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:58 2022] PUT /identity/v3/projects/82fc5a79fc054d4ca04ff73f39b40889/users/69eb16f5d6bd4815af02f5471809d2c7/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 134 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:58.501740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.502043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.502325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.562198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] MFA Rules not processed for user `69eb16f5d6bd4815af02f5471809d2c7`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:58.563520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:58.584841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=4,keystone:SELECT=91,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:58.633781 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:58.739406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 867/1723] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:58 2022] POST /identity/v3/auth/tokens => generated 3323 bytes in 238 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:00:58.755406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.761573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:58.765222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.773471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.773798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.774051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.774842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.787687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.845684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 857/1724] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:58 2022] GET /identity/v3/roles => generated 1420 bytes in 391 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:58.849540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.895530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4b83ccd-dfdb-4f02-b342-cbf4a117261f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.906160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.908345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.908735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.909103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.909811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.911595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.945576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 858/1725] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:00:58 2022] GET /identity/v3/roles => generated 1420 bytes in 99 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:00:58.950026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:58.959485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca6f3a93-19fd-4d2e-ba10-d51a41ca8558 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.968454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:58.969153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:58.969439 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:58.969714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:58.979523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be98f90b-a6b7-4787-b2b6-29f69976088b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:58.994153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:58.995804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c877fd-fb0d-46da-9a7d-79b3a27d256e service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:58.999405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:59.000327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:59.000611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:59.000888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] PATH_INFO: `/v3/projects/5aac22b094e04333bb247eafbc53f51e/users/8c8708ebaf224116b9df115fc8713136/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:59.044561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 868/1726] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:00:58 2022] GET /identity/v3/auth/tokens => generated 3323 bytes in 291 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:00:59.050588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] RBAC: Authorizing `identity:create_grant(project_id=5aac22b094e04333bb247eafbc53f51e, user_id=8c8708ebaf224116b9df115fc8713136, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:59.053741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:59.108051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 859/1727] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:00:58 2022] PUT /identity/v3/projects/5aac22b094e04333bb247eafbc53f51e/users/8c8708ebaf224116b9df115fc8713136/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:59.120069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:59.120257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:59.120496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:59.215347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] MFA Rules not processed for user `8c8708ebaf224116b9df115fc8713136`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:59.220357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:59.344515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:59.361872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 869/1728] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:59 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 242 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:00:59.367373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:59.629841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab348bd-e769-46e0-acf6-2f1b1699b9ac admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:59.643820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:59.645371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:59.645737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:59.645931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] PATH_INFO: `/v3/users/000af482c7fb44e385f255123f94db53` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:59.649212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] RBAC: Authorizing `identity:delete_user(user_id=000af482c7fb44e385f255123f94db53)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:59.650892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:59.696060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:59.708343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b273618b-bb0b-496e-9e71-ad48d88bb474 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:00:59.736664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:00:59.737594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:59.737982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:59.738562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:59.739302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:00:59.741388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f25568e-7b8b-4584-a6bf-94c95c293a6b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:00:59.783038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '000af482c7fb44e385f255123f94db53'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:59.803238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '000af482c7fb44e385f255123f94db53'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:59.804627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '000af482c7fb44e385f255123f94db53'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:59.815132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '000af482c7fb44e385f255123f94db53'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:59.839998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '000af482c7fb44e385f255123f94db53'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:00:59.849208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 870/1729] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:00:59 2022] POST /identity/v3/projects => generated 385 bytes in 479 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:00:59.849208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dc1a5-f310-43ea-b5e6-d52c10fc1ea5 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:00:59.849208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dc1a5-f310-43ea-b5e6-d52c10fc1ea5 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:00:59.849208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba6dc1a5-f310-43ea-b5e6-d52c10fc1ea5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:00:59.857181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:00:59.865660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 860/1730] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:00:59 2022] DELETE /identity/v3/users/000af482c7fb44e385f255123f94db53 => generated 0 bytes in 521 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:00:59.867030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:00:59.869910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:00:59.959019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ba6dc1a5-f310-43ea-b5e6-d52c10fc1ea5 None None] MFA Rules not processed for user `8f471531dac249d994f864102f7b9d3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:00:59.962828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ba6dc1a5-f310-43ea-b5e6-d52c10fc1ea5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.163939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 871/1731] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:00:59 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 318 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:00.165471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.165791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.166155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.214999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:00.216983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.217590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ff70d15-5215-4b6d-bd46-1e2b2d008f81 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:00.241653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 872/1732] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:00.242705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:00.244353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.254666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:00.256327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.256702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.258195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.259587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:00.265100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4cc905ce-62fd-48f8-af3e-9e735bf744db admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:00.392719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 861/1733] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:00:59 2022] POST /identity/v3/users => generated 293 bytes in 526 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:00.394360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.396674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.397156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.444628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] MFA Rules not processed for user `283a1bfe0e3b45aaa0a33e5fe6b1d85a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:00.448226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.595334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79dbe6a0-5354-4430-a693-ba7534e4c68f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:00.619921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:00.620621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.620909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.628064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.678736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:00.682211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-804ccafb-05f0-4239-bd1a-989cfe1e1325 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:00.731448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 862/1734] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:00 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 338 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:00.735708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:00.740074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 873/1735] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 498 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:00.746547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.764527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.773582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.774804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.857598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8c1100c-d174-42ea-966e-098112cd3167 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:00.873834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:00.874517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.874849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:00.875143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:00.877638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:00.881000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:00.918722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 874/1736] 10.222.0.39 () {58 vars in 1015 bytes} [Sun Jul 24 01:01:00 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 177 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:00.919687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:00.924819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:00.934386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:00.956470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aaf67b6a-bebf-400c-a5e0-ea36c8bf73d2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:00.981418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:00.982798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:00.985869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 863/1737] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:00 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 252 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:01.006663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:01.009619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:01.016559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:01.023914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:01.026159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:01.028380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:01.045116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-473a6e1c-2bfa-49b1-8ec1-9cd9cf1bf8c6 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:01.059744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:01.077824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:01.078213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:01.080167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:01.117553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 875/1738] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:00 2022] GET /identity/v3/roles => generated 1420 bytes in 195 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:01.120710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:01.134752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:01.146069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:01.173725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e2aafef-dd70-410b-9aaa-179eee192dd6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:01.186874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 864/1739] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:01 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 163 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:01.199279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:01.199991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:01.200263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:01.202144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:01.204109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:01.211458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7487146a-d5d0-4b64-ba73-a29417212fcf admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:01.270641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 876/1740] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:01 2022] GET /identity/v3/roles => generated 1420 bytes in 151 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:01.271863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:01.308351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1121185d-1d14-48bb-a9f0-f6250d7d8462 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:01.330658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:01.334905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:01.344682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:01.344682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] PATH_INFO: `/v3/projects/52c27aee94d74077b8cdbe0eb0751809/users/c6e941f2d7274a0fb28d2e6d207a420c/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:01.447994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] RBAC: Authorizing `identity:create_grant(project_id=52c27aee94d74077b8cdbe0eb0751809, user_id=c6e941f2d7274a0fb28d2e6d207a420c, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:01.450691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:01.492307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:01.507314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 865/1741] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:01 2022] PUT /identity/v3/projects/52c27aee94d74077b8cdbe0eb0751809/users/c6e941f2d7274a0fb28d2e6d207a420c/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 236 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:01.524442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:01.524647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:01.524886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:01.586885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] MFA Rules not processed for user `c6e941f2d7274a0fb28d2e6d207a420c`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:01.593017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:01.623512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:02.003444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 877/1742] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:01 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 481 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:02.008804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:02.136665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:02.139524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:02.337126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65d9a15a-b5d6-4549-9b19-e8a511511218 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:02.345854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:02.350705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:02.351405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:02.351850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:02.352800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:02.355123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:02.445163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 866/1743] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:02 2022] POST /identity/v3/projects => generated 385 bytes in 436 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:02.446263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:02.524313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff2b0dd4-f6e8-44bb-8e4e-c03490c132ca None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:02.712867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1c5fd0e-0b01-4826-9f88-1717ea1e46aa admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:02.734798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:02.735563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:02.735853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:02.736187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:02.744568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:02.748936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:02.760897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:02.763033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:02.764535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:02.764885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] PATH_INFO: `/v3/projects/2a44d084328f4197b505ead7127d47a0` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:02.768637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] RBAC: Authorizing `identity:delete_project(project_id=2a44d084328f4197b505ead7127d47a0)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:02.774607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:02.807953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 867/1744] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:02 2022] POST /identity/v3/users => generated 299 bytes in 362 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:02.832581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:02.912560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-034f9788-da8e-43f2-b01b-41eebd1ac98a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:02.978660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:02.987004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:02.987262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:02.987557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:02.993361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:02.995155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:03.067610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 868/1745] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:02 2022] GET /identity/v3/roles => generated 1420 bytes in 255 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:03.069545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:03.105943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fdbfac28-d8b3-404b-bdc3-44a3b122ef41 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:03.113739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '2a44d084328f4197b505ead7127d47a0', 'request_id': 'req-09987021-a8b7-461c-a297-4f211c64e043'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:03.151761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '2a44d084328f4197b505ead7127d47a0', 'request_id': 'req-09987021-a8b7-461c-a297-4f211c64e043'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:03.181304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:03.201059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '2a44d084328f4197b505ead7127d47a0', 'request_id': 'req-09987021-a8b7-461c-a297-4f211c64e043'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:03.268950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] The token cache is being invalidate because project 2a44d084328f4197b505ead7127d47a0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:03.272644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:03.275710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:03.282222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 878/1746] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:02 2022] DELETE /identity/v3/projects/2a44d084328f4197b505ead7127d47a0 => generated 0 bytes in 1146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:03.289239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:03.291347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:03.432512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:03.439759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:03.443688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:03.444592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] PATH_INFO: `/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5/users/c9e2f32ded174fdca3bbe3872f6b1d18/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:03.474680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:03.513742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=95c9bd62ec6b477d9554f2e7312ad4c5, user_id=c9e2f32ded174fdca3bbe3872f6b1d18, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:03.530544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:03.583404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 869/1747] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:03 2022] PUT /identity/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5/users/c9e2f32ded174fdca3bbe3872f6b1d18/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 503 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:03.586905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:03.603296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:03.730087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09987021-a8b7-461c-a297-4f211c64e043 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:03.971090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d735fe0-c161-4e29-8431-80c57d97acd6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:03.992506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:03.993523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:03.993949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:03.994363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:03.995665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:03.997459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:04.026473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:04.031891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:04.032200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:04.034245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] PATH_INFO: `/v3/users/69eb16f5d6bd4815af02f5471809d2c7` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:04.038553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] RBAC: Authorizing `identity:delete_user(user_id=69eb16f5d6bd4815af02f5471809d2c7)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:04.046545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:04.046875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 870/1748] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:03 2022] GET /identity/v3/roles => generated 1420 bytes in 460 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:04.059362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:04.104601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de10d08f-a5cd-4e92-a712-6d1d38533338 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:04.123626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:04.124708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:04.124997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:04.125278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] PATH_INFO: `/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5/users/c9e2f32ded174fdca3bbe3872f6b1d18/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:04.127899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] RBAC: Authorizing `identity:create_grant(project_id=95c9bd62ec6b477d9554f2e7312ad4c5, user_id=c9e2f32ded174fdca3bbe3872f6b1d18, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:04.136513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:04.182818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 871/1749] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:04 2022] PUT /identity/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5/users/c9e2f32ded174fdca3bbe3872f6b1d18/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:04.182818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:04.182818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:04.197328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '69eb16f5d6bd4815af02f5471809d2c7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.214755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '69eb16f5d6bd4815af02f5471809d2c7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.215737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '69eb16f5d6bd4815af02f5471809d2c7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.243142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '69eb16f5d6bd4815af02f5471809d2c7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.247789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-95586f3e-890d-456c-bf15-f81af1da527f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '69eb16f5d6bd4815af02f5471809d2c7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.328114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 879/1750] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:03 2022] DELETE /identity/v3/users/69eb16f5d6bd4815af02f5471809d2c7 => generated 0 bytes in 1039 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:04.329636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:04.329930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:04.330436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d64ab9d-2ca9-4453-92ee-29b16b733abc admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:04.331374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:04.425601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] MFA Rules not processed for user `c9e2f32ded174fdca3bbe3872f6b1d18`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:04.431046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:04.444753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=11,keystone:SELECT=88,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:04.483115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:04.484172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:04.484577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:04.484867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] PATH_INFO: `/v3/users/8bd9b8d822c643c1bc7cbfceba65842a` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:04.486806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8bd9b8d822c643c1bc7cbfceba65842a)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:04.489387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:04.564551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:04.657082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8bd9b8d822c643c1bc7cbfceba65842a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.683530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8bd9b8d822c643c1bc7cbfceba65842a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.684811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8bd9b8d822c643c1bc7cbfceba65842a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.713681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8bd9b8d822c643c1bc7cbfceba65842a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.755107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 880/1751] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:04 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 426 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:04.760362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-29ba0bce-870c-4bb1-919d-fc9355059119 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8bd9b8d822c643c1bc7cbfceba65842a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:04.773661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:04.776041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:04.808172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 872/1752] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:04 2022] DELETE /identity/v3/users/8bd9b8d822c643c1bc7cbfceba65842a => generated 0 bytes in 631 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:04.813654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:04.815211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:04.815502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:04.887444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:04.888760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:04.982281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96c4d53d-4d73-45dc-940f-7f669778281c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:05.177926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 873/1753] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 367 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:05.254333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:05.255309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:05.255640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:05.256013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:05.407025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:05.408781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:05.505246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:05.507003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:05.535837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 881/1754] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:04 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 763 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:05.679134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8dab293-ba1b-461d-a102-648d59efd186 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:05.718464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:05.718464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:05.718464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:05.731411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:05.770919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:05.772671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:05.825678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 874/1755] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 419 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:06.782269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:06.784501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:06.803107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9649fa91-95c9-432e-a280-a5c4d75e47d4 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:06.819114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:06.819974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:06.820493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:06.820752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:06.823089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:06.845669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:06.962046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:06.964323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9d2a9717-69fb-4fc6-a2bb-c4f5734abbef service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:06.991854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 882/1756] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:06 2022] GET /identity/v3/auth/tokens => generated 3254 bytes in 210 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:07.329799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:07.331289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:07.362970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-925d09f2-15e2-4073-a092-4cab4e0d9e36 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:07.395859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:07.396951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:07.397374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:07.397785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] PATH_INFO: `/v3/projects/82fc5a79fc054d4ca04ff73f39b40889` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:07.399289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] RBAC: Authorizing `identity:delete_project(project_id=82fc5a79fc054d4ca04ff73f39b40889)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:07.401407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:07.424546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63721-da05-4a9f-9104-aa80fe6b0fb9 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:07.425024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63721-da05-4a9f-9104-aa80fe6b0fb9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:07.425413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cb63721-da05-4a9f-9104-aa80fe6b0fb9 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:07.427500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 883/1757] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:01:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:01:07.585353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=10,keystone:SELECT=89,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:07.657478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '82fc5a79fc054d4ca04ff73f39b40889', 'request_id': 'req-68c670fe-a75d-4b4d-93a0-dda346a04f51'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:07.694360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '82fc5a79fc054d4ca04ff73f39b40889', 'request_id': 'req-68c670fe-a75d-4b4d-93a0-dda346a04f51'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:07.723002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '82fc5a79fc054d4ca04ff73f39b40889', 'request_id': 'req-68c670fe-a75d-4b4d-93a0-dda346a04f51'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:07.780297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] The token cache is being invalidate because project 82fc5a79fc054d4ca04ff73f39b40889 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:07.782255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:07.784439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-68c670fe-a75d-4b4d-93a0-dda346a04f51 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:07.790224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 875/1758] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:07 2022] DELETE /identity/v3/projects/82fc5a79fc054d4ca04ff73f39b40889 => generated 0 bytes in 461 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:07.802685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:07.802685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:07.802685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:07.840478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:07.843924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:08.063942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 884/1759] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 270 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:08.074180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.074728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.075244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.159052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:08.165710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:08.196137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.196561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 876/1760] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:08.214173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b1e355f-e1f8-4052-984e-c3f2da07f8ba None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:08.234759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:08.239929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.242591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.245133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.245978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:08.248486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:08.255246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:08.330562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.336156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 885/1761] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:08 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 130 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:08.343737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34114fc3-e77a-4072-8219-8c235052ade0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:08.350740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:08.351660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.352085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.352540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.353706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:08.356848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c07c5786-8abf-4ad3-981e-4cf0665475ee admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:08.419232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.448653 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 877/1762] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:08 2022] POST /identity/v3/projects => generated 377 bytes in 88 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:08.453181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.456827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.457316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.462674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77bb25-1c4b-4e81-ad95-fbb6f7220852 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:08.545270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:08.555729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:08.722051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:08.723077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.723493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.723913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.729117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:08.731293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:08.792905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 886/1763] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:08 2022] POST /identity/v3/users => generated 289 bytes in 374 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:08.794990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.823139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-04e65d88-fc0c-4aab-add8-6e86840c453c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:08.823666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 878/1764] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:01:08 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 370 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:08.837269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:08.838185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.839087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.839087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.839857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:08.847479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:08.891841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 887/1765] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:08.893511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.921473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5cac003-aa4e-404b-95dd-e2fdd7bda99f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:08.933017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:08.933969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:08.934265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:08.934550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:08.939498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:08.941738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-282dc7ac-2da6-410a-bcea-12be916b55e9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:08.974665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 879/1766] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:08 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:08.978931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:08.999317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-355ef2e5-d2bc-47c0-bc2b-70f19bb2bd30 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:09.021528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:09.023130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:09.024685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:09.025164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] PATH_INFO: `/v3/projects/004241cf68b84739bae167c2ce391e81/users/7f6a5df8f9b0471e96a8fec1d61b75fe/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:09.052628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:09.068598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] RBAC: Authorizing `identity:create_grant(project_id=004241cf68b84739bae167c2ce391e81, user_id=7f6a5df8f9b0471e96a8fec1d61b75fe, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:09.074011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:09.133166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 888/1767] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:08 2022] PUT /identity/v3/projects/004241cf68b84739bae167c2ce391e81/users/7f6a5df8f9b0471e96a8fec1d61b75fe/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 151 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:09.139825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:09.140165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:09.141357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:09.217819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] MFA Rules not processed for user `7f6a5df8f9b0471e96a8fec1d61b75fe`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:09.219287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:09.455232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 880/1768] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:09 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 317 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:09.467976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:09.475532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:09.680281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:09.681825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:09.688302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=5,keystone:SELECT=93,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:09.744522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8096da61-23b5-4ec5-b604-aa664e17015b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:09.751023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:09.752967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:09.753538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:09.754673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:09.813366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:09.820787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:09.842464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2f9a40e-ca43-41dc-871d-d220855cca60 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:09.851341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 889/1769] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:09 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 383 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:09.855245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:09.857470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:09.857782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:09.858180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:09.916726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:09.920400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:09.975477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 881/1770] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:09 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 296 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:09.996291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:09.998467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:10.162164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea9cc205-1e61-4284-8392-c60ff5a9b05a service glance] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:10.174143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:10.177149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:10.178148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:10.178446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] PATH_INFO: `/v3/projects/34ab3c8f771f40b6b9c2598ab037ed7e` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:10.181666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] RBAC: Authorizing `identity:delete_project(project_id=34ab3c8f771f40b6b9c2598ab037ed7e)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:10.187015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:10.340797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '34ab3c8f771f40b6b9c2598ab037ed7e', 'request_id': 'req-01980aff-6164-4ad5-863c-c5598580da86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.366132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '34ab3c8f771f40b6b9c2598ab037ed7e', 'request_id': 'req-01980aff-6164-4ad5-863c-c5598580da86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.379284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '34ab3c8f771f40b6b9c2598ab037ed7e', 'request_id': 'req-01980aff-6164-4ad5-863c-c5598580da86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.418569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:10.440737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] The token cache is being invalidate because project 34ab3c8f771f40b6b9c2598ab037ed7e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:10.441118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.442510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-01980aff-6164-4ad5-863c-c5598580da86 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.445532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 890/1771] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:09 2022] DELETE /identity/v3/projects/34ab3c8f771f40b6b9c2598ab037ed7e => generated 0 bytes in 450 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:10.450364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:10.468812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d94221ea-e265-4296-a4a0-eab01600e8ee service nova] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:10.694435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:10.695868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:10.697269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:10.699081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] PATH_INFO: `/v3/users/7236f76bc19845ee8cd115c11f1088a4` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:10.700911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=7236f76bc19845ee8cd115c11f1088a4)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:10.705062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:10.950978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7236f76bc19845ee8cd115c11f1088a4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.966729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7236f76bc19845ee8cd115c11f1088a4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.967622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7236f76bc19845ee8cd115c11f1088a4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:10.980013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7236f76bc19845ee8cd115c11f1088a4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:11.017688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ac64d1d4-11e5-42b7-b73e-5392ede9bc2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7236f76bc19845ee8cd115c11f1088a4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:11.046999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 882/1772] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:10 2022] DELETE /identity/v3/users/7236f76bc19845ee8cd115c11f1088a4 => generated 0 bytes in 597 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:12.302225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:12.303463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:12.304028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:12.309852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 891/1773] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:01:12 2022] GET /identity => generated 270 bytes in 10 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:01:12.380355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:12.381634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:12.382069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:12.435969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:12.437914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:12.450008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:12.584377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 883/1774] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:01:12 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 205 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:12.622644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:12.624545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:12.641213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:12.642860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:12.656473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-37f99e4a-688f-45bf-b826-c70ff43ab144 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:12.683770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:12.684152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:12.684445 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:12.684723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:12.699051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:12.700874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10c2264c-b9d1-4a8d-a820-ce9761ffed43 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:12.714912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4987e971-72e4-468e-89ea-c0f367dceed3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:12.719940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:12.720897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:12.721334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:12.721761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:12.731730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:12.739416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 892/1775] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:12 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:12.819626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:12.821655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:12.842796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 884/1776] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:12 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:13.118028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f7d55-3eae-4364-8b16-dd26b6cd48df None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.118863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f7d55-3eae-4364-8b16-dd26b6cd48df None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.119384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20f7d55-3eae-4364-8b16-dd26b6cd48df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.154993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c20f7d55-3eae-4364-8b16-dd26b6cd48df None None] MFA Rules not processed for user `8f471531dac249d994f864102f7b9d3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:13.158040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c20f7d55-3eae-4364-8b16-dd26b6cd48df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:13.177483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:13.180896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:13.331348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 893/1777] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:13 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 214 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:13.424121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:13.443443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84cf8b4-97af-4c54-929c-f19799c0db61 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:13.454479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:13.455439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.455727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.456337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] PATH_INFO: `/v3/projects/89c86c44547f4bc5b7d217b380008c74` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.457662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] RBAC: Authorizing `identity:delete_project(project_id=89c86c44547f4bc5b7d217b380008c74)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:13.461612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:13.531487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '89c86c44547f4bc5b7d217b380008c74', 'request_id': 'req-d97d882e-ccc3-4345-ad7c-9f40e024417b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:13.546493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '89c86c44547f4bc5b7d217b380008c74', 'request_id': 'req-d97d882e-ccc3-4345-ad7c-9f40e024417b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:13.557513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '89c86c44547f4bc5b7d217b380008c74', 'request_id': 'req-d97d882e-ccc3-4345-ad7c-9f40e024417b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:13.580662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] The token cache is being invalidate because project 89c86c44547f4bc5b7d217b380008c74 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:13.581164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:13.582951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d97d882e-ccc3-4345-ad7c-9f40e024417b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:13.586875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 885/1778] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:13 2022] DELETE /identity/v3/projects/89c86c44547f4bc5b7d217b380008c74 => generated 0 bytes in 409 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:13.594817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.595063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.595297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.635624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:13.637326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:13.790048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 894/1779] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 196 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:13.797940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.798674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.800018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.840205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:13.841817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:13.852426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 886/1780] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 55 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:13.856374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:13.889733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9326d296-70d8-4c3e-91ff-5ac82a56de08 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:13.898856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:13.901447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.903342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.905141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.906423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:13.907667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:13.912746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:13.938403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 895/1781] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:13.943119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:13.955632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-209bab3e-16e6-434f-b924-963e0a912ae9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:13.971969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:13.973263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:13.973625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:13.973816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:13.974578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:13.976925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.060883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 887/1782] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:13 2022] POST /identity/v3/projects => generated 385 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:14.063569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:14.076262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-856fcb80-53ff-456a-bf6e-0f2bd9cdb1cc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:14.248370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:14.249647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.250563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.250563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.251118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:14.263291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.307995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 896/1783] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:14 2022] POST /identity/v3/users => generated 293 bytes in 242 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:14.309724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:14.324182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5639704-2497-40cc-944a-6f9260d6c666 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:14.332748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:14.333493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.333719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.333992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.335104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:14.337261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.375506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 888/1784] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 66 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:14.379865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:14.407578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ef490e4-e373-4b21-9001-bb8c64c3f9ae admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:14.434411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:14.436238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.436753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.437245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.438197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:14.441197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce78899-1881-47ee-88bd-d1e1fe1bb091 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.444545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:14.474480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 897/1785] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:14 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:14.478437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:14.498205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-70f8199f-f226-4bf1-8cf5-947a207f9e59 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:14.510571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:14.511782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.512213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.512660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] PATH_INFO: `/v3/projects/a8a52767984141e59ebe5be98b3f2f1e/users/971d2af89f434d338e749553e22e231a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.543852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a8a52767984141e59ebe5be98b3f2f1e, user_id=971d2af89f434d338e749553e22e231a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:14.547564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.585553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 889/1786] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:14 2022] PUT /identity/v3/projects/a8a52767984141e59ebe5be98b3f2f1e/users/971d2af89f434d338e749553e22e231a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:14.599108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.599399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.599805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.649680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] MFA Rules not processed for user `971d2af89f434d338e749553e22e231a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:14.651160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:14.796235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 898/1787] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:14 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 200 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:14.808655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:14.810673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:14.916676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d65cbff5-63f3-464b-99a5-8411fc31fe48 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:14.921761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:14.922614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:14.922614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:14.923050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:14.933690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:14.935201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5f071f18-269f-4a49-9924-56a92ca9e3b4 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:14.947871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 890/1788] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:14 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:14.970302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:15.152951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d17e87a-39ff-4c2f-b15b-a9b928b5e1d8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:15.166829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:15.168044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:15.168499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:15.168931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] PATH_INFO: `/v3/users/971d2af89f434d338e749553e22e231a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:15.171460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] RBAC: Authorizing `identity:delete_user(user_id=971d2af89f434d338e749553e22e231a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:15.174599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:15.323883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '971d2af89f434d338e749553e22e231a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:15.334737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '971d2af89f434d338e749553e22e231a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:15.336684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '971d2af89f434d338e749553e22e231a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:15.356518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '971d2af89f434d338e749553e22e231a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:15.357282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '971d2af89f434d338e749553e22e231a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:15.389238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:15.389533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:15.389819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:15.400919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 899/1789] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:14 2022] DELETE /identity/v3/users/971d2af89f434d338e749553e22e231a => generated 0 bytes in 413 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:15.416388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:15.418484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:15.472816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:15.593969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 891/1790] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:15 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 207 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:15.604975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:15.607613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:15.633009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:15.731711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f4b36616-b305-49e6-b0a8-dc5beb0463b0 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:15.736893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:15.737884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:15.738232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:15.738577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:15.752086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:15.754800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-454a5ccd-241d-41a1-82a0-9bc5fac22ded service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:15.775792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 900/1791] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:15 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 172 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:16.031279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:16.039527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:16.052398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389ae36d-0396-43bf-be2b-9d81522cefce None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:16.058062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:16.059172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:16.059519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:16.059882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] PATH_INFO: `/v3/users/8c8708ebaf224116b9df115fc8713136` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:16.061185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c8708ebaf224116b9df115fc8713136)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:16.063287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:16.162309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c8708ebaf224116b9df115fc8713136'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:16.173552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c8708ebaf224116b9df115fc8713136'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:16.174948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c8708ebaf224116b9df115fc8713136'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:16.196448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c8708ebaf224116b9df115fc8713136'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:16.219021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c8708ebaf224116b9df115fc8713136'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:16.230529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 892/1792] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:16 2022] DELETE /identity/v3/users/8c8708ebaf224116b9df115fc8713136 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:16.235426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15cb96-7951-4cdf-8221-87164f5a3e65 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:16.236051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15cb96-7951-4cdf-8221-87164f5a3e65 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:16.236051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce15cb96-7951-4cdf-8221-87164f5a3e65 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:16.269685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ce15cb96-7951-4cdf-8221-87164f5a3e65 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:16.271139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ce15cb96-7951-4cdf-8221-87164f5a3e65 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:16.408411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 901/1793] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:16 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:16.423613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:16.426381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:16.530572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbb21b67-22e9-450a-b3a8-1fad45015979 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:16.538037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:16.538954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:16.539606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:16.540123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:16.553967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:16.556033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9e9fca-d558-4283-8cea-7d9b7048be20 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:16.571155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 893/1794] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:16 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 148 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:16.626117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:16.626652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:16.627130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:16.628828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 902/1795] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:01:16 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:01:16.640521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:16.641271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:16.641584 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:16.702910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:16.704249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:16.766929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:16.871893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 894/1796] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:01:16 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 232 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:17.241228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.243399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.267621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0371b061-6a0d-4cb0-9ea2-e4a59cfc48a8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.275474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.279309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.279914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.280415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.293687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.297528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.339920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 903/1797] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:17 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:17.588982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.590451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.602093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.603686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.621998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159d79f0-8268-4f1e-b763-9a872d4c6cad service nova] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.623071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4970bed6-4ee3-4806-9e7d-8fc8e2c15103 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.626951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.627820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.628108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.628546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] PATH_INFO: `/v3/projects/a8a52767984141e59ebe5be98b3f2f1e` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.628939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.629692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a8a52767984141e59ebe5be98b3f2f1e)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.629939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.630291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.630634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] PATH_INFO: `/v3/projects/5aac22b094e04333bb247eafbc53f51e` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.631650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.632636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5aac22b094e04333bb247eafbc53f51e)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.635447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.710970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5aac22b094e04333bb247eafbc53f51e', 'request_id': 'req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.714435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a8a52767984141e59ebe5be98b3f2f1e', 'request_id': 'req-0420869d-04a9-4f89-b62b-ad8634716660'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.720453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5aac22b094e04333bb247eafbc53f51e', 'request_id': 'req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.722809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a8a52767984141e59ebe5be98b3f2f1e', 'request_id': 'req-0420869d-04a9-4f89-b62b-ad8634716660'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.728766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5aac22b094e04333bb247eafbc53f51e', 'request_id': 'req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.731087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a8a52767984141e59ebe5be98b3f2f1e', 'request_id': 'req-0420869d-04a9-4f89-b62b-ad8634716660'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.744440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] The token cache is being invalidate because project 5aac22b094e04333bb247eafbc53f51e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:17.744751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.745963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.747825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 895/1798] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:17 2022] DELETE /identity/v3/projects/5aac22b094e04333bb247eafbc53f51e => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:17.752140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.754093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.755693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] The token cache is being invalidate because project a8a52767984141e59ebe5be98b3f2f1e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:17.755996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.757341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0420869d-04a9-4f89-b62b-ad8634716660 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:17.759845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 904/1799] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:17 2022] DELETE /identity/v3/projects/a8a52767984141e59ebe5be98b3f2f1e => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:17.764592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d143c63-3e71-4d02-9fd0-920a441ad9c4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.764887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d143c63-3e71-4d02-9fd0-920a441ad9c4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.765165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9d143c63-3e71-4d02-9fd0-920a441ad9c4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.784088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9d143c63-3e71-4d02-9fd0-920a441ad9c4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:17.785339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9d143c63-3e71-4d02-9fd0-920a441ad9c4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.804328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:17.832649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b306b34a-ce7e-4ac8-8032-ed1fd300a7d7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.867221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 905/1800] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:17.874511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.874834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.875126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.900896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:17.902107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:17.914194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 906/1801] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:17.915348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.915501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.916291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.916570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.916840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] PATH_INFO: `/v3/users/c6e941f2d7274a0fb28d2e6d207a420c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.918113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6e941f2d7274a0fb28d2e6d207a420c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.923026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.929038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f4ec967-f87f-43dd-9b1d-ef04f766aa75 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.929677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:17.935451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.936203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.936586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.936970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.937882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:17.938169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.939949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.953189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 907/1802] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:17 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:17.958190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:17.970384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-011d2312-a1d1-47e3-98b9-0f42f3fb5c95 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:17.978588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:17.979317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:17.979586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:17.979866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:17.980565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:17.982867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:17.991898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6e941f2d7274a0fb28d2e6d207a420c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.008197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6e941f2d7274a0fb28d2e6d207a420c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.009124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6e941f2d7274a0fb28d2e6d207a420c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.017845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6e941f2d7274a0fb28d2e6d207a420c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.019538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 908/1803] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:17 2022] POST /identity/v3/projects => generated 373 bytes in 61 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:18.023525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.026031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:18.032986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6e941f2d7274a0fb28d2e6d207a420c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.040724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 896/1804] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:17 2022] DELETE /identity/v3/users/c6e941f2d7274a0fb28d2e6d207a420c => generated 0 bytes in 289 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:18.109973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fea6b9-95ce-4d0e-9bd0-a2586c1d897a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.176082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.177112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.177395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.177672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.178517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.181105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.208027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 909/1805] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:18 2022] POST /identity/v3/users => generated 287 bytes in 181 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:18.209683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.228816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-957ade84-d766-4bcb-baba-346eeef34ba1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.246107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.246826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.247122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.247404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.248149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.249764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.277057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 897/1806] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:18 2022] GET /identity/v3/roles => generated 1420 bytes in 68 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:18.281027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.297102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3082859-2f1c-4399-a1cf-19b895209397 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.309642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.310494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.310830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.311129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.311937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.320938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18975f56-94d5-49e7-b270-64e6261651e6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.351566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 910/1807] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:18 2022] GET /identity/v3/roles => generated 1420 bytes in 71 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:18.357982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.371305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a335fa9-ab27-463d-96ac-1d9b2385ed36 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.376240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.377145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.377433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.377715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] PATH_INFO: `/v3/projects/8eb42ca1fefa47e0a65ad2fb75d408a6/users/dafa9370cd7c41e1a3110a0aee1037f8/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.403604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8eb42ca1fefa47e0a65ad2fb75d408a6, user_id=dafa9370cd7c41e1a3110a0aee1037f8, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.415200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.447680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 898/1808] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:18 2022] PUT /identity/v3/projects/8eb42ca1fefa47e0a65ad2fb75d408a6/users/dafa9370cd7c41e1a3110a0aee1037f8/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 90 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:18.458492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.458805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.459144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.476073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:18.506849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] MFA Rules not processed for user `dafa9370cd7c41e1a3110a0aee1037f8`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:18.508129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:18.567540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.570960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:18.618483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 911/1809] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:18 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:18.621032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.624486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:18.727239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc53cdeb-44c9-4e03-86aa-a3b2a9df3e90 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.739741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.740650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.740940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.741221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] PATH_INFO: `/v3/projects/52c27aee94d74077b8cdbe0eb0751809` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.743645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] RBAC: Authorizing `identity:delete_project(project_id=52c27aee94d74077b8cdbe0eb0751809)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.745880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.757886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d424b923-148a-4d22-baf7-deaf07a2e3bd None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.763748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.764980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.765121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.765703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.782124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.785968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.797404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 912/1810] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:18 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 177 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:18.813816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:18.817005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '52c27aee94d74077b8cdbe0eb0751809', 'request_id': 'req-e67699a5-5e13-4787-8c95-2236949d98dc'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.827038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '52c27aee94d74077b8cdbe0eb0751809', 'request_id': 'req-e67699a5-5e13-4787-8c95-2236949d98dc'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.832560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.839089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '52c27aee94d74077b8cdbe0eb0751809', 'request_id': 'req-e67699a5-5e13-4787-8c95-2236949d98dc'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.854802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038e127b-81fa-4840-8474-722e56c1b36c service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.857689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] The token cache is being invalidate because project 52c27aee94d74077b8cdbe0eb0751809 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:18.857995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.858911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:18.860797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 899/1811] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:18 2022] DELETE /identity/v3/projects/52c27aee94d74077b8cdbe0eb0751809 => generated 0 bytes in 295 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:18.867366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:18.872552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:18.940444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:18.943329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:18.946928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:18.947235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] PATH_INFO: `/v3/users/dafa9370cd7c41e1a3110a0aee1037f8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:18.949416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] RBAC: Authorizing `identity:delete_user(user_id=dafa9370cd7c41e1a3110a0aee1037f8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:18.952539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:18.976287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e67699a5-5e13-4787-8c95-2236949d98dc admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:18.999649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:19.000602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:19.001027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:19.001701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] PATH_INFO: `/v3/users/c9e2f32ded174fdca3bbe3872f6b1d18` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:19.004312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] RBAC: Authorizing `identity:delete_user(user_id=c9e2f32ded174fdca3bbe3872f6b1d18)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:19.006811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:19.024648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:19.035633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'dafa9370cd7c41e1a3110a0aee1037f8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.053489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'dafa9370cd7c41e1a3110a0aee1037f8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.054063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'dafa9370cd7c41e1a3110a0aee1037f8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.065410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'dafa9370cd7c41e1a3110a0aee1037f8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.066824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1a343f8a-6bb7-4349-9310-d4d92875101e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'dafa9370cd7c41e1a3110a0aee1037f8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.084557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c9e2f32ded174fdca3bbe3872f6b1d18'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.090722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 913/1812] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:18 2022] DELETE /identity/v3/users/dafa9370cd7c41e1a3110a0aee1037f8 => generated 0 bytes in 259 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:19.095270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:19.095825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:19.096333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:19.099358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c9e2f32ded174fdca3bbe3872f6b1d18'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.100366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c9e2f32ded174fdca3bbe3872f6b1d18'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.109586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c9e2f32ded174fdca3bbe3872f6b1d18'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.125923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:19.127404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:19.136034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c9e2f32ded174fdca3bbe3872f6b1d18'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:19.146647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 900/1813] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:18 2022] DELETE /identity/v3/users/c9e2f32ded174fdca3bbe3872f6b1d18 => generated 0 bytes in 276 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:19.225592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 914/1814] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:19.269258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:19.271583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:19.365942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7dbe623a-0d72-482c-9da8-b179cd4c65ac admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:19.377472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:19.379122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:19.379843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:19.380269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:19.407351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:19.497480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:19.499083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58a0015c-0b66-495f-869f-288c6adfa8b5 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:19.513847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 901/1815] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 244 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:20.259938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:20.261589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:20.279554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf9208eb-3551-44ef-98b5-611307d261ef None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:20.299499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:20.301588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.301588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.301825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] PATH_INFO: `/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.308807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] RBAC: Authorizing `identity:delete_project(project_id=95c9bd62ec6b477d9554f2e7312ad4c5)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:20.311218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:20.430493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '95c9bd62ec6b477d9554f2e7312ad4c5', 'request_id': 'req-1db840d6-e7d4-499d-9d6c-e79ab8770011'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:20.444291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '95c9bd62ec6b477d9554f2e7312ad4c5', 'request_id': 'req-1db840d6-e7d4-499d-9d6c-e79ab8770011'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:20.467816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '95c9bd62ec6b477d9554f2e7312ad4c5', 'request_id': 'req-1db840d6-e7d4-499d-9d6c-e79ab8770011'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:20.488603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] The token cache is being invalidate because project 95c9bd62ec6b477d9554f2e7312ad4c5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:20.488603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:20.489601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1db840d6-e7d4-499d-9d6c-e79ab8770011 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:20.499922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 915/1816] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:20 2022] DELETE /identity/v3/projects/95c9bd62ec6b477d9554f2e7312ad4c5 => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:20.500495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.500942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.501064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.529732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:20.531425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:20.630822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 902/1817] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:20.640495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.640923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.641917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.669662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:20.670932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:20.683752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 916/1818] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:20.688485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:20.698995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cd7a965-df04-48a2-8c96-404dc9eda3b5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:20.703806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:20.704490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.704782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.705062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.705830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:20.706139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:20.708486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:20.721753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 903/1819] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:20.726293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:20.737555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9625e74-d5db-4d7c-9301-8355e4874600 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:20.743250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:20.743928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.744216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.744494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.745196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:20.747164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:20.779324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 917/1820] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:20 2022] POST /identity/v3/projects => generated 401 bytes in 53 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:20.783473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:20.796248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cf8cab7-9254-426c-99b8-64aa3652a0bb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:20.902280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:20.903203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.903636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.904142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.914116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:20.916749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:20.959199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 904/1821] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:20 2022] POST /identity/v3/users => generated 301 bytes in 176 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:20.964756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:20.980200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6653486-4bf5-40e5-9074-93817b4cbb62 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:20.988595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:20.990123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:20.990629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:20.991703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:20.992732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:20.995679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.006460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:21.019390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 918/1822] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:20 2022] GET /identity/v3/roles => generated 1420 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:21.021149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.039647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cf513f7-c40b-428f-8973-aeda0eefa260 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.045525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.046361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.046854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.047122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.048075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.050174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.067126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.069352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:21.098133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 905/1823] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 77 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:21.099220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.130192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3458fb83-fd70-4220-8ab4-7545454a7d49 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.136677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-557d2f76-eddc-44e0-a8b7-fcb361ce6845 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.145301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.146552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.147087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.147390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.147507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.147674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.147936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] PATH_INFO: `/v3/projects/8eb42ca1fefa47e0a65ad2fb75d408a6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.148038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] PATH_INFO: `/v3/projects/939b8a81878841eea0888ff06a2a88bb/users/c6b5f9da0eaf4d51a05caf133ad34f7c/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.149611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] RBAC: Authorizing `identity:delete_project(project_id=8eb42ca1fefa47e0a65ad2fb75d408a6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.156151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.157938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:21.167121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] RBAC: Authorizing `identity:create_grant(project_id=939b8a81878841eea0888ff06a2a88bb, user_id=c6b5f9da0eaf4d51a05caf133ad34f7c, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.171088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1827e66-0b54-410c-b8dd-5e7a121b62ae admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.202815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 906/1824] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:21 2022] PUT /identity/v3/projects/939b8a81878841eea0888ff06a2a88bb/users/c6b5f9da0eaf4d51a05caf133ad34f7c/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:21.217783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.218077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.218349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.234570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8eb42ca1fefa47e0a65ad2fb75d408a6', 'request_id': 'req-8f1327fb-8676-4fb1-93bf-691c6c29669e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:21.258047 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8eb42ca1fefa47e0a65ad2fb75d408a6', 'request_id': 'req-8f1327fb-8676-4fb1-93bf-691c6c29669e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:21.265506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] MFA Rules not processed for user `c6b5f9da0eaf4d51a05caf133ad34f7c`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:21.267560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8eb42ca1fefa47e0a65ad2fb75d408a6', 'request_id': 'req-8f1327fb-8676-4fb1-93bf-691c6c29669e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:21.277000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:21.295906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] The token cache is being invalidate because project 8eb42ca1fefa47e0a65ad2fb75d408a6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:21.295991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:21.296810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f1327fb-8676-4fb1-93bf-691c6c29669e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:21.298753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 919/1825] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:21 2022] DELETE /identity/v3/projects/8eb42ca1fefa47e0a65ad2fb75d408a6 => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:21.306468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6c9d8-eed1-4dd6-baa5-1f69d0227855 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.306883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6c9d8-eed1-4dd6-baa5-1f69d0227855 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.307645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bb6c9d8-eed1-4dd6-baa5-1f69d0227855 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.354728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7bb6c9d8-eed1-4dd6-baa5-1f69d0227855 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:21.356767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7bb6c9d8-eed1-4dd6-baa5-1f69d0227855 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:21.481698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 907/1826] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3268 bytes in 261 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:21.484021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.493954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:21.577093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 920/1827] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 272 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:21.580692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7e3f00bd-1b00-4f02-9bf1-afdc2f11c526 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.586727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.587056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.587341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.588690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.589999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.590571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.591211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.592130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.594904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.611208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:21.612774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:21.627859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 921/1828] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 42 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:21.629263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.633921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 908/1829] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/projects => generated 401 bytes in 150 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:21.638253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.639776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5109ae2-f519-4c63-9847-29feb079b638 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.716785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.717764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.718115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.718972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.719994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:21.720366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.723461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.739989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 922/1830] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:21 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 111 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:21.745159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.768263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c613ba74-5fa1-4291-a99b-4a4a3cd5b761 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.784522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.785328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.786334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.786723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.787565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.790369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.809002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:21.839652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce079a12-e669-4dcf-bdb4-58b8d192e902 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:21.849924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 923/1831] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/projects => generated 381 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:21.854385 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:21.855219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:21.855540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:21.855670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:21.856351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:21.856469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.859359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:21.868339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:21.887840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 909/1832] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/users => generated 307 bytes in 250 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:21.892907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:21.905406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dffa140f-7d3b-4430-b33d-2727aa06d8ab admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.004735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f977f357-71fc-4655-a71f-f5f3371c9316 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.009811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.010604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.011072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.011270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.011562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.012263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.012526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.012839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.013266 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.014096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.014556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.016664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.028111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 910/1833] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:21 2022] GET /identity/v3/roles => generated 1420 bytes in 137 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:22.032532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.050199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 924/1834] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:21 2022] POST /identity/v3/users => generated 291 bytes in 195 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:22.054007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.054700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-738aa211-a381-45da-88ca-b233745b0208 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.061728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.062873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.063160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.063503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] PATH_INFO: `/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b/users/de0ca215a1c14d09bef8efc33971d1f6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.069762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-50d69054-4310-4e69-b98f-46c67142adb1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.077317 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.078406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.078932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.079415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.080356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.083434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.087254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ffc1544fd82457c89a7bcecbb57ac9b, user_id=de0ca215a1c14d09bef8efc33971d1f6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.090299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.100374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 925/1835] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/roles => generated 1420 bytes in 47 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:22.104181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.119109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 911/1836] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:22 2022] PUT /identity/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b/users/de0ca215a1c14d09bef8efc33971d1f6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 84 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:22.122878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4df3ac72-450f-4b8a-803d-6873a76519a2 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.127661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.209847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.210969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.211469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.211994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.213126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.215359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.222238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc088536-62c8-4c08-8849-cd0d51fced4e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.230093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.230816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.231100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.231519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.232263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.233339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 926/1837] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/roles => generated 1420 bytes in 129 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:22.236207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.239888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.264602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e3f5faf-8070-4f87-a136-2839c574e0b1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.273304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 912/1838] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/roles => generated 1420 bytes in 146 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:22.274000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.311586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.312503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.312790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.313122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] PATH_INFO: `/v3/projects/4dd26bb2797345d6b531f5ba5db5cbc8/users/0e4b9bb864f24c93b2c9547432b8fd96/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.324944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1221bba-f877-451d-8ed4-0ea9c97395b7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.337753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.338669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.338985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.339268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] PATH_INFO: `/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b/users/de0ca215a1c14d09bef8efc33971d1f6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.342995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] RBAC: Authorizing `identity:create_grant(project_id=0ffc1544fd82457c89a7bcecbb57ac9b, user_id=de0ca215a1c14d09bef8efc33971d1f6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.345272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4dd26bb2797345d6b531f5ba5db5cbc8, user_id=0e4b9bb864f24c93b2c9547432b8fd96, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.345836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-112cdd98-3bf1-48d6-b67a-0f90f6c95f2b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.349421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9d955c9-73ee-496d-84ed-ccefee7e0d03 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.365413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 913/1839] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:22 2022] PUT /identity/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b/users/de0ca215a1c14d09bef8efc33971d1f6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 92 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:22.379122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.379575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.379870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.390080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 927/1840] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:22 2022] PUT /identity/v3/projects/4dd26bb2797345d6b531f5ba5db5cbc8/users/0e4b9bb864f24c93b2c9547432b8fd96/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:22.391938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.392242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.392510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.411710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] MFA Rules not processed for user `de0ca215a1c14d09bef8efc33971d1f6`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:22.413093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:22.448481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:22.450485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:INSERT=4,keystone:SELECT=96 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:22.451186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:22.520667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:22.549899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 914/1841] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:22 2022] POST /identity/v3/auth/tokens => generated 3335 bytes in 176 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:22.564856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 928/1842] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 174 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:22.565464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.567486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:22.577029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.579050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:22.630669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eeb999ac-db0b-48de-a254-e2f1ac32ac98 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.634573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.635253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.635536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.635812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.639893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f95a0fc-d681-4cba-85e2-3624cf206fb3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.643661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.645458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.645742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.646016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.646354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.647817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.656510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.658522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa22cad4-d699-41ab-b731-9ec8e763f76f service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.658810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 915/1843] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3335 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:22.670906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 929/1844] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:22.737699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:22.747723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c447fa2d-c5ae-4346-a7bf-f0025ac92101 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:22.753939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:22.754551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:22.754947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:22.755256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:22.852045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:22.853841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:22.876576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 916/1845] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:22 2022] GET /identity/v3/auth/tokens => generated 3268 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:23.908908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14ac3f8-dc7f-42a5-b203-7b2c08c6d2b1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:23.909985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14ac3f8-dc7f-42a5-b203-7b2c08c6d2b1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:23.910128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14ac3f8-dc7f-42a5-b203-7b2c08c6d2b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:23.947252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e14ac3f8-dc7f-42a5-b203-7b2c08c6d2b1 None None] MFA Rules not processed for user `0e4b9bb864f24c93b2c9547432b8fd96`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:23.948519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e14ac3f8-dc7f-42a5-b203-7b2c08c6d2b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:24.031179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 930/1846] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:23 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:24.041915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:24.051569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d7b6582-4590-4472-9be2-5d5ba3e634e3 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:24.055974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:24.056709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:24.057089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:24.057375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:24.069267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:24.071261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:24.081893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 917/1847] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:24 2022] GET /identity/v3/auth/tokens => generated 3248 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:25.570234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6910918-0003-4a7f-a835-ef0639630cd0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:25.570863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6910918-0003-4a7f-a835-ef0639630cd0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:25.570980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6910918-0003-4a7f-a835-ef0639630cd0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:25.589415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f6910918-0003-4a7f-a835-ef0639630cd0 None None] MFA Rules not processed for user `0e4b9bb864f24c93b2c9547432b8fd96`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:25.590730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f6910918-0003-4a7f-a835-ef0639630cd0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:25.599859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 931/1848] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:25 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:27.251491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:27.266424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0a77a65-158f-4e20-aa32-06ef82b255c9 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:27.272019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:27.273039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.273426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.273946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] PATH_INFO: `/v3/users/c6b5f9da0eaf4d51a05caf133ad34f7c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.275428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] RBAC: Authorizing `identity:delete_user(user_id=c6b5f9da0eaf4d51a05caf133ad34f7c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:27.277472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:27.333948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c6b5f9da0eaf4d51a05caf133ad34f7c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.343135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c6b5f9da0eaf4d51a05caf133ad34f7c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.343766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c6b5f9da0eaf4d51a05caf133ad34f7c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.345162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:27.357011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c6b5f9da0eaf4d51a05caf133ad34f7c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.400758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c6b5f9da0eaf4d51a05caf133ad34f7c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.410027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 918/1849] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:27 2022] DELETE /identity/v3/users/c6b5f9da0eaf4d51a05caf133ad34f7c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:27.415879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.416939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.417530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.442950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:27.445516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:27.496345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:27.497860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:27.541182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 932/1850] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 127 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:27.553144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:27.554660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:27.564362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:27.573005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afa0543b-be4e-442c-80ad-11f4a05397ce admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:27.577879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:27.578932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.579375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.579756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] PATH_INFO: `/v3/users/7f6a5df8f9b0471e96a8fec1d61b75fe` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.582002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] RBAC: Authorizing `identity:delete_user(user_id=7f6a5df8f9b0471e96a8fec1d61b75fe)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:27.584091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:27.620115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a78cb421-e0bb-47aa-a65c-5dd646927a0b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:27.624406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:27.625198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.625483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.625767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.641059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:27.642883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e70d032f-0981-4edc-8019-9da2306cdde1 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:27.645974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7f6a5df8f9b0471e96a8fec1d61b75fe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.657824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7f6a5df8f9b0471e96a8fec1d61b75fe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.658973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7f6a5df8f9b0471e96a8fec1d61b75fe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.661408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 933/1851] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:27.672182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7f6a5df8f9b0471e96a8fec1d61b75fe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.686190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7f6a5df8f9b0471e96a8fec1d61b75fe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:27.694144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 919/1852] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:27 2022] DELETE /identity/v3/users/7f6a5df8f9b0471e96a8fec1d61b75fe => generated 0 bytes in 198 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:27.698548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.699096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.699694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.719752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:27.720919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:27.790004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 934/1853] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:27.799937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:27.801779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:27.872853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e8c5bacd-3c30-4f6b-b6b6-bd65c76071ea admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:27.879665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:27.880016 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:27.880309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:27.880589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:27.888312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:27.892616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:27.895757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:27.910974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 920/1854] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:27.980311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:27.981836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:28.011503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bd27f2e-5b50-4100-9bf5-a77d3e08482c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.036601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.037576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.037780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.038099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] PATH_INFO: `/v3/projects/939b8a81878841eea0888ff06a2a88bb` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.039393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] RBAC: Authorizing `identity:delete_project(project_id=939b8a81878841eea0888ff06a2a88bb)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.041479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.148777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '939b8a81878841eea0888ff06a2a88bb', 'request_id': 'req-3fe5c48f-b911-4d2c-8515-9691295e581a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.162156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '939b8a81878841eea0888ff06a2a88bb', 'request_id': 'req-3fe5c48f-b911-4d2c-8515-9691295e581a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.170057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '939b8a81878841eea0888ff06a2a88bb', 'request_id': 'req-3fe5c48f-b911-4d2c-8515-9691295e581a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.187292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] The token cache is being invalidate because project 939b8a81878841eea0888ff06a2a88bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:28.187589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.188544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.194204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 935/1855] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:27 2022] DELETE /identity/v3/projects/939b8a81878841eea0888ff06a2a88bb => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:28.195314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.216479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d8f5a63a-50d2-4577-80f9-ed6cacd7d412 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.285330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.286224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.286812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.287104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] PATH_INFO: `/v3/users/de0ca215a1c14d09bef8efc33971d1f6` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.287785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.288347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] RBAC: Authorizing `identity:delete_user(user_id=de0ca215a1c14d09bef8efc33971d1f6)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.289209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:28.290465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.335232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3fe5c48f-b911-4d2c-8515-9691295e581a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.351233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'de0ca215a1c14d09bef8efc33971d1f6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.351589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.353712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.354002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.355028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] PATH_INFO: `/v3/projects/004241cf68b84739bae167c2ce391e81` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.356393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] RBAC: Authorizing `identity:delete_project(project_id=004241cf68b84739bae167c2ce391e81)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.358805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.361685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'de0ca215a1c14d09bef8efc33971d1f6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.363353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'de0ca215a1c14d09bef8efc33971d1f6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.372371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'de0ca215a1c14d09bef8efc33971d1f6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.388068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6b4d7551-d368-41a9-b61e-d85b8a6434e1 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'de0ca215a1c14d09bef8efc33971d1f6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.398467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 921/1856] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:28 2022] DELETE /identity/v3/users/de0ca215a1c14d09bef8efc33971d1f6 => generated 0 bytes in 204 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:28.404904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:28.433491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '004241cf68b84739bae167c2ce391e81', 'request_id': 'req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.442695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '004241cf68b84739bae167c2ce391e81', 'request_id': 'req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.451682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '004241cf68b84739bae167c2ce391e81', 'request_id': 'req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.469533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] The token cache is being invalidate because project 004241cf68b84739bae167c2ce391e81 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:28.470094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.471009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ddc724af-4dbc-40ce-94e7-9955f97bbc8b admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.474588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 936/1857] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:28 2022] DELETE /identity/v3/projects/004241cf68b84739bae167c2ce391e81 => generated 0 bytes in 187 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:28.480627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.481214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.481894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.521710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:28.522977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:28.607484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 922/1858] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:28.608962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.609251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.609527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.637941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:28.639442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:28.661231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 937/1859] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 54 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:28.665574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.681240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f95da083-d707-42ca-a718-5c91f6472215 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.689184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.689545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.692541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.692849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.693615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:28.694183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.702209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.718451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:28.728276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 923/1860] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:28.730008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.746142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc333d18-46a3-481f-8efb-9159bf719554 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.755338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.756940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.758381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.772436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.773340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.775469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.809838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.821140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:28.834846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e04eefa5-031b-4b1a-8a8a-d85d251a0f0d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.838816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 938/1861] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:28 2022] POST /identity/v3/projects => generated 369 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:28.841522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.908882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.909873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.910255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.910542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] PATH_INFO: `/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.912787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0ffc1544fd82457c89a7bcecbb57ac9b)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.914737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.914866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-739d8ac5-16a9-4d26-9421-c2c37fa5cf37 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.919783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:28.920600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:28.920889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:28.921288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:28.922155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:28.924162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:28.951424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 939/1862] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:28 2022] POST /identity/v3/users => generated 285 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:28.954838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:28.965854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb15514d-351c-410a-bb77-b5fc1c0afaff admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:28.975739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0ffc1544fd82457c89a7bcecbb57ac9b', 'request_id': 'req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.982309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0ffc1544fd82457c89a7bcecbb57ac9b', 'request_id': 'req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:28.991507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0ffc1544fd82457c89a7bcecbb57ac9b', 'request_id': 'req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:29.006481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] The token cache is being invalidate because project 0ffc1544fd82457c89a7bcecbb57ac9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:29.006800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:29.007500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d39cf6e7-5a36-4ce8-a8a6-e3744bd08545 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:29.009200 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 924/1863] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:28 2022] DELETE /identity/v3/projects/0ffc1544fd82457c89a7bcecbb57ac9b => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:29.014367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72b213-a149-4562-b7da-efb229447445 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.014663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72b213-a149-4562-b7da-efb229447445 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.015082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d72b213-a149-4562-b7da-efb229447445 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.034627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.035380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.035515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3d72b213-a149-4562-b7da-efb229447445 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:29.035658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.035925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.036653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.036758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3d72b213-a149-4562-b7da-efb229447445 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.038265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.044872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 925/1864] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:29.053966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 940/1865] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:28 2022] GET /identity/v3/roles => generated 1420 bytes in 97 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:29.054314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.054314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.055053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.055253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.056859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.070645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-245462a1-6f1a-4f58-9c55-51913b429c9a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.075181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:29.075521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.076188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.076376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.076484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.076741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.077465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.079174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.084841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 926/1866] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:29.088801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.089582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 941/1867] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:29.093345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.098315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=5,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:29.098485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d98ba0e-347e-4200-b9a8-e1a1eadfe126 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.103593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.104251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.104542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.104827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.104939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-945b9809-1741-4c78-97cd-1c12504fbc6c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.105561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:29.105941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.107518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.110600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.111598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.111887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.112690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] PATH_INFO: `/v3/projects/ee2ddfe60895421e940fdae99604890f/users/75c780dab9e2416aabc0ce84829308b8/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.120129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 927/1868] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:01:29.124566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.132911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee2ddfe60895421e940fdae99604890f, user_id=75c780dab9e2416aabc0ce84829308b8, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.134684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a257407f-b208-4608-a3b1-25ddc2dab1f3 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.135762 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8c81b829-520e-44de-8f76-4bb57f1281b2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.139463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.139989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.140290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.140673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.141475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.143723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.149385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 942/1869] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:29 2022] PUT /identity/v3/projects/ee2ddfe60895421e940fdae99604890f/users/75c780dab9e2416aabc0ce84829308b8/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:29.162941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.163272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.163746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.180968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 928/1870] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/projects => generated 409 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:29.185001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.193716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] MFA Rules not processed for user `75c780dab9e2416aabc0ce84829308b8`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:29.195229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.236555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:29.259322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-722fb4d1-8aab-48fd-9814-125e7780c109 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.263990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.264658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.264959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.265421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.265941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.268166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.270172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 943/1871] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 108 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:29.285333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.287076 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.305752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 929/1872] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/users => generated 305 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:29.310430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.337403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccb66189-3c21-4744-8710-9c1e75271bbf admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.346253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.347277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.347751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.348372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.350516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.353822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.356780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbdc48b3-c372-4289-887e-be4418a62a08 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.361503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.362561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.362952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.363421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.370820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 930/1873] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/roles => generated 1420 bytes in 61 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:29.375133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.376556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.378722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.388229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53d7ca70-b1d1-4156-bfde-cd6fe24cb6b0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.390491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 944/1874] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 106 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:29.404716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.405763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.406451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.407014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.408547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.410456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05d3bb40-8e61-44df-971a-824b2e901cbc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.427182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 931/1875] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/roles => generated 1420 bytes in 53 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:29.431632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.445798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58c9f1d0-3db3-49f4-aed9-91154325a39e service glance] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.451779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.452785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.453073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.453406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] PATH_INFO: `/v3/projects/33f13149333847a283244ffd8a32fdb6/users/6bd6398fe50f41128883e7dc541c6594/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.472364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=33f13149333847a283244ffd8a32fdb6, user_id=6bd6398fe50f41128883e7dc541c6594, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.475287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.490310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 945/1876] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:29 2022] PUT /identity/v3/projects/33f13149333847a283244ffd8a32fdb6/users/6bd6398fe50f41128883e7dc541c6594/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:29.501884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.502182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.502469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.540318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] MFA Rules not processed for user `6bd6398fe50f41128883e7dc541c6594`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:29.542916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:29.639946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 932/1877] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/auth/tokens => generated 3276 bytes in 139 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:29.645135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.701334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:29.718525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b1626e8-4985-4fe4-aed7-a6fb96a376c3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.723967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.724665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.725029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.725272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.726095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.728007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.763418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 946/1878] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/projects => generated 407 bytes in 117 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:29.764960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.779054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e71c238-c030-401d-818f-aee440374fb5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.865813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.866751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.867110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.867423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.868206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.871342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.895921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 933/1879] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:29 2022] POST /identity/v3/users => generated 310 bytes in 132 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:29.909787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.925490 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1922b6f3-c6d5-457e-9b79-16ed60fdfc5e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.931933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.932117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.932391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.932668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:29.933392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:29.935584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:29.949796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 947/1880] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:29 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:29.953885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:29.966375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fed41201-9280-487a-bfde-7974891eebc1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:29.972743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:29.974027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:29.974444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:29.974951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] PATH_INFO: `/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55/users/abd8c99327464f978e4c572424f48caf/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.011713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe17cb2ba0e94c22b42e4a286047ea55, user_id=abd8c99327464f978e4c572424f48caf, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:30.011713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:30.023051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 934/1881] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:29 2022] PUT /identity/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55/users/abd8c99327464f978e4c572424f48caf/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 69 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:30.026708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:30.045309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56025a5f-82a0-4b91-a85d-81d1ccf003dc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:30.122400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:30.123116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:30.123398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:30.123704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.124427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:30.126761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a36b39c4-3ed6-44ba-a038-618bbb2a7c02 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:30.135961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 948/1882] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:30 2022] GET /identity/v3/roles => generated 1420 bytes in 110 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:30.140178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:30.147014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:30.152089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85b49a9f-a57d-4a30-94e1-d4e280a9faeb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:30.158510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:30.160184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:30.160797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:30.162426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] PATH_INFO: `/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55/users/abd8c99327464f978e4c572424f48caf/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.164042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe17cb2ba0e94c22b42e4a286047ea55, user_id=abd8c99327464f978e4c572424f48caf, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:30.167040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:30.180134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 935/1883] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:30 2022] PUT /identity/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55/users/abd8c99327464f978e4c572424f48caf/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:30.190343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:30.190758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:30.191273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.218966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] MFA Rules not processed for user `abd8c99327464f978e4c572424f48caf`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:30.220469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:30.310000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 949/1884] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:30 2022] POST /identity/v3/auth/tokens => generated 3341 bytes in 117 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:30.318800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:30.320920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:30.390632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3916f691-d5a4-4d6d-bd20-2e64a35b1b59 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:30.397968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:30.398630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:30.398949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:30.399225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.416655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:30.417536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:30.431494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 936/1885] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:30 2022] GET /identity/v3/auth/tokens => generated 3341 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:30.597124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:30.607011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ab2f2c2-9275-46f3-ac99-48f5ac6f34d1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:30.610519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:30.611277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:30.611601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:30.611937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:30.665588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:30.686985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:30.688536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-459c49bf-9e03-47ed-85fc-385263752d40 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:30.698024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 950/1886] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:30 2022] GET /identity/v3/auth/tokens => generated 3276 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:39.376923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:39.465633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2715a723-0646-49c7-ad0d-1264b45f9914 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:39.468422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:39.469449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:39.469732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:39.470008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] PATH_INFO: `/v3/users/6bd6398fe50f41128883e7dc541c6594` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:39.471302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6bd6398fe50f41128883e7dc541c6594)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:39.473827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:39.555174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6bd6398fe50f41128883e7dc541c6594'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:39.568000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6bd6398fe50f41128883e7dc541c6594'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:39.568651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6bd6398fe50f41128883e7dc541c6594'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:39.578522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6bd6398fe50f41128883e7dc541c6594'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:39.601681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6bd6398fe50f41128883e7dc541c6594'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:39.612392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 937/1887] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:39 2022] DELETE /identity/v3/users/6bd6398fe50f41128883e7dc541c6594 => generated 0 bytes in 235 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:39.621920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:39.623300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:39.623751 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:39.645992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:39.647246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:39.728142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 951/1888] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:39.738174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:39.739643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:39.760108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:39.825358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a1b17a5-2762-4afd-8437-23242d5ec7c6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:39.829083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:39.829745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:39.830032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:39.830314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:39.840741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:39.842193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:39.853209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 938/1889] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:39 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 116 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:40.159838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:40.161349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:40.173394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-64086303-6ada-495c-8e70-c79856a43e4b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:40.178268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:40.179165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:40.179455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:40.179735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] PATH_INFO: `/v3/projects/33f13149333847a283244ffd8a32fdb6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:40.180944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=33f13149333847a283244ffd8a32fdb6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:40.182990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:40.233214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '33f13149333847a283244ffd8a32fdb6', 'request_id': 'req-6a560f79-1be9-4c87-a00d-b133ab0782a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.240620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '33f13149333847a283244ffd8a32fdb6', 'request_id': 'req-6a560f79-1be9-4c87-a00d-b133ab0782a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.246756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '33f13149333847a283244ffd8a32fdb6', 'request_id': 'req-6a560f79-1be9-4c87-a00d-b133ab0782a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.257930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] The token cache is being invalidate because project 33f13149333847a283244ffd8a32fdb6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:40.258218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.258928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.260497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 952/1890] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:40 2022] DELETE /identity/v3/projects/33f13149333847a283244ffd8a32fdb6 => generated 0 bytes in 102 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:40.264504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:40.274749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9defc4f5-4995-4fc2-be9c-62044e0573fb service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:40.333312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:40.334126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:40.334407 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:40.334695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] PATH_INFO: `/v3/users/abd8c99327464f978e4c572424f48caf` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:40.335891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] RBAC: Authorizing `identity:delete_user(user_id=abd8c99327464f978e4c572424f48caf)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:40.337809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:40.383384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'abd8c99327464f978e4c572424f48caf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.390832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'abd8c99327464f978e4c572424f48caf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.391466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'abd8c99327464f978e4c572424f48caf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.399188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'abd8c99327464f978e4c572424f48caf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.411579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db14fcd3-3986-4152-a32d-597adb05c165 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'abd8c99327464f978e4c572424f48caf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.418215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 939/1891] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:40 2022] DELETE /identity/v3/users/abd8c99327464f978e4c572424f48caf => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:40.793222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:40.795128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:40.818381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a560f79-1be9-4c87-a00d-b133ab0782a0 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:40.884675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:40.885555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:40.885840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:40.886118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] PATH_INFO: `/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:40.887571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe17cb2ba0e94c22b42e4a286047ea55)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:40.891494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:40.899219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:40.943401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe17cb2ba0e94c22b42e4a286047ea55', 'request_id': 'req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.952010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe17cb2ba0e94c22b42e4a286047ea55', 'request_id': 'req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.958759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe17cb2ba0e94c22b42e4a286047ea55', 'request_id': 'req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.969927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] The token cache is being invalidate because project fe17cb2ba0e94c22b42e4a286047ea55 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:40.970298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.971190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-feb67ef6-e80c-4cbc-b2a0-75a6370bf11f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:40.973270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 953/1892] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:40 2022] DELETE /identity/v3/projects/fe17cb2ba0e94c22b42e4a286047ea55 => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:40.978219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:40.978518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:40.978819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:40.999972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:41.001179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:41.036436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:41.068842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 940/1893] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:41.078343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.078627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.078964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.098230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:41.099444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:41.107546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 954/1894] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:41.111649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.125795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5486e193-8316-46a4-9b94-db9a348bf36e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.131695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.132377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.132667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.132950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.133721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:41.134034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.135683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.147983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 941/1895] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:41 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:41.152353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.161944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-42236aaf-55db-46ad-8368-7ca4f9a71348 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.166660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.167361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.167648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.167925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.168622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.170653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.202742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 955/1896] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/projects => generated 367 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:41.204045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.214126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0409f3c-4558-4d7d-911d-0fc5d87f739f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.277416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.278095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.278383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.278661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.279391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.281314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.302194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 942/1897] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/users => generated 284 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:41.306510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.316853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b97c6673-196a-41c0-a341-528d15705996 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.321714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.322372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.322658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.322969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.323727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.325341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.336329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 956/1898] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:41.340467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.351275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-de9dfd19-30e2-4270-9200-0aeadc0d3756 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.356024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.356679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.356965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.357239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.357952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.359579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-772b4732-dfea-4920-936a-a8ed30948bee admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.370246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 943/1899] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:41.374155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.384981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e82276b8-0d42-4015-aaf6-72494b90bad6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.390171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.391178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.391465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.391816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] PATH_INFO: `/v3/projects/a9f486348de04d04a17c1bdc1e617ac5/users/0724128726bb4357a45562711ea99b63/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.409607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9f486348de04d04a17c1bdc1e617ac5, user_id=0724128726bb4357a45562711ea99b63, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.412471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.426296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 957/1900] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:41 2022] PUT /identity/v3/projects/a9f486348de04d04a17c1bdc1e617ac5/users/0724128726bb4357a45562711ea99b63/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:41.437909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.438380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.438479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.466086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] MFA Rules not processed for user `0724128726bb4357a45562711ea99b63`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:41.467332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:41.535728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 944/1901] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:41.540285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.601976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2599adb-e9b0-4aac-bae2-d0d66b159c3d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.606575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.607265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.607553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.607824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.608497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.610404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.636529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 958/1902] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/projects => generated 367 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:41.640525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.651707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e494c8f-22a6-4225-8805-14345039f77f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.681255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:41.710997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.711673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.711961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.712240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.712937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.714892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.744722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 945/1903] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/users => generated 290 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:41.746004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.756942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83966532-bf45-4703-a364-4a28f70c4c63 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.762365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.763101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.763436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.763775 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.764564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.766207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.777373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 959/1904] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:41.781537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.792150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce227b0a-9b7d-47e1-9510-68e898ab7d3c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.796785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.797667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.797951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.798227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] PATH_INFO: `/v3/projects/dec2f27aa0394bf99839c528d6156448/users/95db12d92cd144669078bc7c0be6b97f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.814052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] RBAC: Authorizing `identity:create_grant(project_id=dec2f27aa0394bf99839c528d6156448, user_id=95db12d92cd144669078bc7c0be6b97f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.816819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.832807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 946/1905] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:41 2022] PUT /identity/v3/projects/dec2f27aa0394bf99839c528d6156448/users/95db12d92cd144669078bc7c0be6b97f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:41.836771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.843298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:41.848880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd77b6da-fdc3-4013-a27c-9196b8f67926 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.917811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.918552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.918788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.919082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.919818 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.921421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7648306e-414e-4b70-8da4-32d20a1857be admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.934402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 960/1906] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:41 2022] GET /identity/v3/roles => generated 1420 bytes in 95 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:41.935624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:41.946564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0c8788e-9bb1-421d-8e6c-6ce431e38726 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:41.954679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:41.955629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.956432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.956824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] PATH_INFO: `/v3/projects/dec2f27aa0394bf99839c528d6156448/users/95db12d92cd144669078bc7c0be6b97f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:41.959192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] RBAC: Authorizing `identity:create_grant(project_id=dec2f27aa0394bf99839c528d6156448, user_id=95db12d92cd144669078bc7c0be6b97f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:41.961866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:41.973673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 947/1907] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:41 2022] PUT /identity/v3/projects/dec2f27aa0394bf99839c528d6156448/users/95db12d92cd144669078bc7c0be6b97f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:41.983757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:41.984050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:41.984332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:42.011090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] MFA Rules not processed for user `95db12d92cd144669078bc7c0be6b97f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:42.012301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:42.086874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 961/1908] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:41 2022] POST /identity/v3/auth/tokens => generated 3301 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:42.099785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:42.101699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:42.162569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d1c8681-91e3-407e-a11f-1ae976a3ea0d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:42.166237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:42.167001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:42.167289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:42.167565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:42.178102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:42.179582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:42.189807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 948/1909] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:42 2022] GET /identity/v3/auth/tokens => generated 3301 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:42.281395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:42.295450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2139e891-3de2-484d-913f-6fd4d5d3e2b2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:42.299123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:42.299794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:42.300116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:42.300394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:42.362597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:42.364305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b9e42ad-6187-4800-a0f0-a47f79ec6f4a service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:42.374558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 962/1910] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:42 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:47.466314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:47.529145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-490afd96-4b46-449e-8a96-cff6c671f7de service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:47.533547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:47.535541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:47.535822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:47.536094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] PATH_INFO: `/v3/users/0724128726bb4357a45562711ea99b63` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:47.537579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] RBAC: Authorizing `identity:delete_user(user_id=0724128726bb4357a45562711ea99b63)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:47.539545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:47.568555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:INSERT=5,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:47.588679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0724128726bb4357a45562711ea99b63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:47.595940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0724128726bb4357a45562711ea99b63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:47.597717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0724128726bb4357a45562711ea99b63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:47.604754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0724128726bb4357a45562711ea99b63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:47.617161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0724128726bb4357a45562711ea99b63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:47.623061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 949/1911] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:47 2022] DELETE /identity/v3/users/0724128726bb4357a45562711ea99b63 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:47.627563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:47.628347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:47.628634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:47.646370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:47.647606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:47.668598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:47.714448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 963/1912] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:47 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:47.725473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:47.728188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:47.812055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ae70ebb-3b4d-4073-8399-dc3ec6dac58e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:47.816511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:47.817160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:47.817443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:47.817716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:47.826988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:47.828344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:47.838596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 950/1913] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:47 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:48.232873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:48.234852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:48.249559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4b7344e-3b9d-4d99-8914-630f23eb7856 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:48.257269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:48.258831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:48.259357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:48.259931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] PATH_INFO: `/v3/projects/a9f486348de04d04a17c1bdc1e617ac5` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:48.261850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9f486348de04d04a17c1bdc1e617ac5)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:48.264274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:48.320236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9f486348de04d04a17c1bdc1e617ac5', 'request_id': 'req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.328717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9f486348de04d04a17c1bdc1e617ac5', 'request_id': 'req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.335733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9f486348de04d04a17c1bdc1e617ac5', 'request_id': 'req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.348339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] The token cache is being invalidate because project a9f486348de04d04a17c1bdc1e617ac5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:48.348626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.349305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.354332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 964/1914] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:48 2022] DELETE /identity/v3/projects/a9f486348de04d04a17c1bdc1e617ac5 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:48.358240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:48.369451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d917ec4f-f00f-4832-96f6-f5307be1594a service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:48.460986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:48.461979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:48.462260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:48.462800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] PATH_INFO: `/v3/users/95db12d92cd144669078bc7c0be6b97f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:48.463823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=95db12d92cd144669078bc7c0be6b97f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:48.465738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:48.528807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '95db12d92cd144669078bc7c0be6b97f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.534706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:48.540211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '95db12d92cd144669078bc7c0be6b97f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.541372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '95db12d92cd144669078bc7c0be6b97f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.553102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '95db12d92cd144669078bc7c0be6b97f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.572329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9daac366-2bc9-461e-85f5-d83072852f3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '95db12d92cd144669078bc7c0be6b97f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:48.581856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 951/1915] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:48 2022] DELETE /identity/v3/users/95db12d92cd144669078bc7c0be6b97f => generated 0 bytes in 224 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:48.882272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:48.883908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:48.897312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86fb3846-4c71-4d89-b8ac-0187eb6d1ce9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:48.956467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:48.957333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:48.957617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:48.957891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] PATH_INFO: `/v3/projects/dec2f27aa0394bf99839c528d6156448` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:48.959108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=dec2f27aa0394bf99839c528d6156448)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:48.961040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.005799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'dec2f27aa0394bf99839c528d6156448', 'request_id': 'req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:49.007437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:49.013431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'dec2f27aa0394bf99839c528d6156448', 'request_id': 'req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:49.020030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'dec2f27aa0394bf99839c528d6156448', 'request_id': 'req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:49.030930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] The token cache is being invalidate because project dec2f27aa0394bf99839c528d6156448 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:49.031214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:49.031842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-60bc25aa-6697-419a-9b13-bc6dc4f73eb1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:49.033373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 965/1916] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:48 2022] DELETE /identity/v3/projects/dec2f27aa0394bf99839c528d6156448 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:49.038950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.039246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.039523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.057142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:49.058331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:49.116085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 952/1917] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:49.124791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.125143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.125391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.145422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:49.146639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:49.156062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 966/1918] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:49.160755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.170879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88734d27-35f2-47a2-b725-c466a7982fc6 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.180843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.182024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.182487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.183017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.184201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:49.184519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.186163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.199920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 953/1919] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:49 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:49.204244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.217609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cfc42db-8ca1-4811-9fa6-b6421965c6da None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.224115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.225302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.225778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.226237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.227175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.229786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.262230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 967/1920] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/projects => generated 385 bytes in 56 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:49.263868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.281064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54497455-0026-4f8d-aed6-989a4286ad07 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.356402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.357365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.357770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.358237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.359033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.361633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.389547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 954/1921] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/users => generated 293 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:49.394362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.404877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8ffa4d29-cd5b-4e1d-a506-07a1941f99b7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.417171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.417906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.418336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.418623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.419402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.421000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.434153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 968/1922] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:49.438329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.448749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd3bbd8e-219e-4afe-88f1-61fdd8e20a73 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.453365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.454021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.454305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.454582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.455337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.457465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a1e1bdf-dbc1-42df-8d63-d5cb2798969f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.468099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 955/1923] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:49.472720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.482758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bbb719-3896-4c7a-9645-94dea17f8bec admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.487660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.488559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.488841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.489117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] PATH_INFO: `/v3/projects/81c295ec881a47148b7c357198f8a5be/users/31f05d5f46814b378869b729e7919206/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.504492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=81c295ec881a47148b7c357198f8a5be, user_id=31f05d5f46814b378869b729e7919206, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.507203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.520679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 969/1924] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:49 2022] PUT /identity/v3/projects/81c295ec881a47148b7c357198f8a5be/users/31f05d5f46814b378869b729e7919206/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:49.531107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.531641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.531755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.559965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] MFA Rules not processed for user `31f05d5f46814b378869b729e7919206`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:49.561276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:49.583401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:49.633277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 956/1925] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:49.637778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.715316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aab05de-10d9-4fee-886e-176e812959d1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.720026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.720628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.720908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.721183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.721906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.723894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.750539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 970/1926] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/projects => generated 387 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:49.755201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.765771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d4008ee7-a17f-4f2b-910b-7e747249305a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.825962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.826672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.827003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.827286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.827986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.829943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.852493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 957/1927] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:49 2022] POST /identity/v3/users => generated 300 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:49.858075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.870588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aad4160-759c-4dfe-a481-d649517d1fe6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.875693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.876367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.876658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.876939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.878279 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.879927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.890897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 971/1928] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:49.895055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.905194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4af209f5-68ae-4811-a8a1-d2807c9923a8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:49.910438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:49.911486 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:49.911776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:49.912059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] PATH_INFO: `/v3/projects/b6470ca285fc428db8123a2de68dc6f4/users/f624acdac53842ee8c8fe04ebe58a47b/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:49.929576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6470ca285fc428db8123a2de68dc6f4, user_id=f624acdac53842ee8c8fe04ebe58a47b, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:49.932380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:49.946985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 958/1929] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:49 2022] PUT /identity/v3/projects/b6470ca285fc428db8123a2de68dc6f4/users/f624acdac53842ee8c8fe04ebe58a47b/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:49.950941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:49.961225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9670c4b2-d3da-4112-a9d6-3a674b2a26c2 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:50.006460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=4,keystone:DELETE=3,keystone:SELECT=93 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:50.019701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:50.020371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:50.020650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:50.020924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:50.021641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:50.023283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fef880-331f-48f3-88a5-137fd811cd1b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:50.033381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 972/1930] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:49 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:50.037759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:50.048127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22b59264-45de-40d8-b9eb-51567cc2aae6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:50.053979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:50.054905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:50.055192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:50.055613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] PATH_INFO: `/v3/projects/b6470ca285fc428db8123a2de68dc6f4/users/f624acdac53842ee8c8fe04ebe58a47b/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:50.057668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6470ca285fc428db8123a2de68dc6f4, user_id=f624acdac53842ee8c8fe04ebe58a47b, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:50.060381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:50.071995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 959/1931] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:50 2022] PUT /identity/v3/projects/b6470ca285fc428db8123a2de68dc6f4/users/f624acdac53842ee8c8fe04ebe58a47b/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:50.082991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:50.083290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:50.083571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:50.110084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] MFA Rules not processed for user `f624acdac53842ee8c8fe04ebe58a47b`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:50.111339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:50.184890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 973/1932] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:50 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:50.198426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:50.200355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:50.270181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d1f80c5-4d4e-4dd9-918f-ca91a04ba9af admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:50.273755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:50.274403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:50.274686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:50.274996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:50.285084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:50.286565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd89211c-4676-4a3f-8975-30a224eb2d8f service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:50.296159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 960/1933] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:50 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 98 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:50.932937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:50.997739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ef0d554-af77-4c6e-b261-0c0fea8c8473 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:51.003092 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:51.004094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:51.004373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:51.004769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] PATH_INFO: `/v3/users/31f05d5f46814b378869b729e7919206` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:51.006168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] RBAC: Authorizing `identity:delete_user(user_id=31f05d5f46814b378869b729e7919206)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:51.008555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:51.059849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '31f05d5f46814b378869b729e7919206'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.076034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '31f05d5f46814b378869b729e7919206'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.076666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '31f05d5f46814b378869b729e7919206'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.089256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '31f05d5f46814b378869b729e7919206'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.090013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '31f05d5f46814b378869b729e7919206'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.100725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:51.126510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 974/1934] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:50 2022] DELETE /identity/v3/users/31f05d5f46814b378869b729e7919206 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:51.131461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:51.131970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:51.132250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:51.152501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:51.153817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:51.170821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:51.254067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 961/1935] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:51 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:51.264134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:51.265905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:51.344091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b6443e1-b570-4d57-aa38-5910f30b1602 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:51.348039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:51.348697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:51.348980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:51.349255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:51.359543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:51.361129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:51.372450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 975/1936] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:51.739891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:51.742885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:51.765006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-623a74bc-8f52-4572-9b89-79b97467d5f0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:51.772467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:51.773482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:51.773880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:51.774318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] PATH_INFO: `/v3/projects/81c295ec881a47148b7c357198f8a5be` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:51.776635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] RBAC: Authorizing `identity:delete_project(project_id=81c295ec881a47148b7c357198f8a5be)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:51.779448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:51.848558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '81c295ec881a47148b7c357198f8a5be', 'request_id': 'req-00c0d5d5-6628-42c4-8705-ef269f51e752'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.859549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '81c295ec881a47148b7c357198f8a5be', 'request_id': 'req-00c0d5d5-6628-42c4-8705-ef269f51e752'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.869771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '81c295ec881a47148b7c357198f8a5be', 'request_id': 'req-00c0d5d5-6628-42c4-8705-ef269f51e752'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.885243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] The token cache is being invalidate because project 81c295ec881a47148b7c357198f8a5be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:51.885558 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.886562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:51.890823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 962/1937] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:51 2022] DELETE /identity/v3/projects/81c295ec881a47148b7c357198f8a5be => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:51.892436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:51.903940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2eaacb4-823c-4fd5-84cc-b06826ce6203 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:51.974109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:51.975319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:51.975674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:51.976033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] PATH_INFO: `/v3/users/f624acdac53842ee8c8fe04ebe58a47b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:51.977690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] RBAC: Authorizing `identity:delete_user(user_id=f624acdac53842ee8c8fe04ebe58a47b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:51.979801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:52.035344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f624acdac53842ee8c8fe04ebe58a47b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.045265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f624acdac53842ee8c8fe04ebe58a47b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.046173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f624acdac53842ee8c8fe04ebe58a47b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.056137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f624acdac53842ee8c8fe04ebe58a47b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.057144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c79a1d0e-22d7-4b13-84c7-246605c75b58 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f624acdac53842ee8c8fe04ebe58a47b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.082084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 976/1938] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:51 2022] DELETE /identity/v3/users/f624acdac53842ee8c8fe04ebe58a47b => generated 0 bytes in 190 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:52.536771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:52.538327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:52.567374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c0d5d5-6628-42c4-8705-ef269f51e752 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:52.637665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:52.638542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:52.638860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:52.639166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] PATH_INFO: `/v3/projects/b6470ca285fc428db8123a2de68dc6f4` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:52.640362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6470ca285fc428db8123a2de68dc6f4)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:52.642288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:52.698551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:52.703325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6470ca285fc428db8123a2de68dc6f4', 'request_id': 'req-258cf06d-e3dd-47c2-b77e-5d86531044ba'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.712857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6470ca285fc428db8123a2de68dc6f4', 'request_id': 'req-258cf06d-e3dd-47c2-b77e-5d86531044ba'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.724359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6470ca285fc428db8123a2de68dc6f4', 'request_id': 'req-258cf06d-e3dd-47c2-b77e-5d86531044ba'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.741575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] The token cache is being invalidate because project b6470ca285fc428db8123a2de68dc6f4 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:52.751077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.751077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-258cf06d-e3dd-47c2-b77e-5d86531044ba admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:52.751077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 963/1939] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:52 2022] DELETE /identity/v3/projects/b6470ca285fc428db8123a2de68dc6f4 => generated 0 bytes in 208 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:52.755107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:52.755801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:52.756226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:52.781874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:52.791971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:52.798482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:52.934857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 977/1940] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 182 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:52.943468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:52.943758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:52.945405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:52.964082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:52.965371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:52.975242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 964/1941] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:52 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:52.978921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:52.989376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61dd6877-66ee-4a87-b6b8-1f7111ccc391 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:52.994155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:52.995177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:52.995457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:52.995931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:52.996672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:52.997135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:52.998788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.014962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 978/1942] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:52 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:53.019743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.029791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e90102a2-c0bc-4ee6-a466-c27630316539 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.035583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.036265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.036556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.036841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.037564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.039547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.069696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 965/1943] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:53 2022] POST /identity/v3/projects => generated 383 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:53.074064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.090595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0889e1dd-1566-4e08-910f-5dca1c8a0b47 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.183557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.184415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.184545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.185324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.185958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.188061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.210708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 979/1944] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:53 2022] POST /identity/v3/users => generated 292 bytes in 137 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:53.214896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.227485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71edc8e9-c36b-4810-a9c6-0692fcef1066 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.233095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.233829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.234122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.234404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.235207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.236847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.249396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 966/1945] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:53.253947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.265935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17d245ac-78eb-44bf-9338-5e6af30d3c82 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.271939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.272626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.272981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.273274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.274091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.276303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9505838-ea39-47e7-b5c7-9f556356ce1d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.288871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 980/1946] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:53.292997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.303931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb9b771c-bb98-46e2-892b-bce36f367e3c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.310459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.311461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.311745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.312019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] PATH_INFO: `/v3/projects/77db5d77e9f74d8980c9516e732f5855/users/e32e398f4c5b43a28da845a71344183a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.341766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] RBAC: Authorizing `identity:create_grant(project_id=77db5d77e9f74d8980c9516e732f5855, user_id=e32e398f4c5b43a28da845a71344183a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.344519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.364101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 967/1947] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:53 2022] PUT /identity/v3/projects/77db5d77e9f74d8980c9516e732f5855/users/e32e398f4c5b43a28da845a71344183a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:53.376276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.376725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.377155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.441910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] MFA Rules not processed for user `e32e398f4c5b43a28da845a71344183a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:53.443307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:53.578263 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 981/1948] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:53 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 203 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:53.583878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.663350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e9180ed-856a-4699-ad7e-4cf12d20a7cd admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.668184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.668858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.669147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.669427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.670132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.672124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.703901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 968/1949] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:53 2022] POST /identity/v3/projects => generated 383 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:53.708160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.729913 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f05ef9-9d8d-45a0-9518-d39fd74b0ffa None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.734350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:53.807211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.808096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.808429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.808805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.809614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.811825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.840792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 982/1950] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:53 2022] POST /identity/v3/users => generated 292 bytes in 133 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:53.846033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.868284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ce5b5e4-7553-461e-bccc-a43d3b323f2f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.874508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.875554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.875722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.876102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.876941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.879065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.909277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 969/1951] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 64 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:53.913497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.928752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae3447d2-8d9c-4faa-8c9e-2942c4df5e0a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.934454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.935164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.935498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.935782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.936534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.938149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-123708c6-1475-43ae-a2ab-75271215ccee admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:53.949069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 983/1952] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:53 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:53.954550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:53.967078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f41a83b7-9c19-47b2-8943-9e5619314b01 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:53.972613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:53.973520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:53.973843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:53.974120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] PATH_INFO: `/v3/projects/f2a3553cb2c04be583bb38753177a2b1/users/70f874188d9a4bbbba8be4074840aed8/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:53.991306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=f2a3553cb2c04be583bb38753177a2b1, user_id=70f874188d9a4bbbba8be4074840aed8, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:53.994113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.007988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 970/1953] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:53 2022] PUT /identity/v3/projects/f2a3553cb2c04be583bb38753177a2b1/users/70f874188d9a4bbbba8be4074840aed8/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 55 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:54.078245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0228589-31da-41d9-968e-317626dc5625 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.078803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0228589-31da-41d9-968e-317626dc5625 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.079107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0228589-31da-41d9-968e-317626dc5625 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.106346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c0228589-31da-41d9-968e-317626dc5625 None None] MFA Rules not processed for user `70f874188d9a4bbbba8be4074840aed8`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:54.107692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c0228589-31da-41d9-968e-317626dc5625 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.174280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 984/1954] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:54 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:54.178830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.199671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=4,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:54.257080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b00260f1-ee5e-4904-a0cd-ea55b79a6dd6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.266551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.267267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.267549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.267826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.269120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.271163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.296183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 971/1955] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:54 2022] POST /identity/v3/projects => generated 385 bytes in 118 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:54.300386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0228589-31da-41d9-968e-317626dc5625 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.311191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0228589-31da-41d9-968e-317626dc5625 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.373659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.373659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.374062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.374169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.374903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.377309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb1dff86-cc19-4c6b-ba6b-c3ea1ab15876 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.399856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 985/1956] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:54 2022] POST /identity/v3/users => generated 299 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:54.403962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.416348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2df28398-bc31-4efe-a3d6-0eaa5a79722b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.423998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.425270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.425654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.426004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.426848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.428531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95685693-97f6-4c67-a596-029853b5d3e4 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.436434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c7b8e-74ee-4490-89e4-ce9967e299da None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.436912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c7b8e-74ee-4490-89e4-ce9967e299da None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.437245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d6c7b8e-74ee-4490-89e4-ce9967e299da None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.438810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 986/1957] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:01:54 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (2 switches on core 0) Jul 24 01:01:54.447938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 972/1958] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:54.448942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6c7b8e-74ee-4490-89e4-ce9967e299da None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.457087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:54.463210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d6c7b8e-74ee-4490-89e4-ce9967e299da None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.468313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.469199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.469485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.469815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] PATH_INFO: `/v3/projects/72ba336f4ac545c6be7e9db47c4d7038/users/8230cacc6a894c2e8d37be25d80e930d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.472358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.472358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.472655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.495307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72ba336f4ac545c6be7e9db47c4d7038, user_id=8230cacc6a894c2e8d37be25d80e930d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.495475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:54.496552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.498579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.522197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 987/1959] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:54 2022] PUT /identity/v3/projects/72ba336f4ac545c6be7e9db47c4d7038/users/8230cacc6a894c2e8d37be25d80e930d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 71 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:54.522197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.527401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.565844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 973/1960] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:01:54 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:54.566894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.578246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00c43722-e7a2-4f0a-8745-e82a16d14755 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.596096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0d9226-4105-4769-aec8-c6734aab6754 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.601365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.602256 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.602541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.602849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] PATH_INFO: `/v3/users/b88e33e967ed47e780b3a1d64c5a0b8a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.604120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b88e33e967ed47e780b3a1d64c5a0b8a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.606069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.641431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.642281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.642564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.642875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.644317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.646028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.657664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 974/1961] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:54 2022] GET /identity/v3/roles => generated 1420 bytes in 90 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:54.658663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.661819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.671652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b88e33e967ed47e780b3a1d64c5a0b8a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:54.680851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b88e33e967ed47e780b3a1d64c5a0b8a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:54.681502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b88e33e967ed47e780b3a1d64c5a0b8a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:54.690434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b88e33e967ed47e780b3a1d64c5a0b8a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:54.691598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b88e33e967ed47e780b3a1d64c5a0b8a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:54.695264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:54.717551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 988/1962] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:54 2022] DELETE /identity/v3/users/b88e33e967ed47e780b3a1d64c5a0b8a => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:54.718572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.720217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.774417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dcf3043-96a3-4130-9776-ad30cdc0e392 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.804570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c061739a-18cd-43b5-9658-b04c798caeb6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.809864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.810777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.811100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.811382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] PATH_INFO: `/v3/projects/72ba336f4ac545c6be7e9db47c4d7038/users/8230cacc6a894c2e8d37be25d80e930d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.813428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=72ba336f4ac545c6be7e9db47c4d7038, user_id=8230cacc6a894c2e8d37be25d80e930d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.816249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7c76303-9ada-4cc6-93d8-e9797ef29ba1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.819938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:54.829934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 989/1963] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:54 2022] PUT /identity/v3/projects/72ba336f4ac545c6be7e9db47c4d7038/users/8230cacc6a894c2e8d37be25d80e930d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 112 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:54.840548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.840840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.841121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.850033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:54.851047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:54.851047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:54.851240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:54.852949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.867246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:54.868887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:54.876104 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] MFA Rules not processed for user `8230cacc6a894c2e8d37be25d80e930d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:54.877377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.881959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 975/1964] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:54 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:54.952891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:54.954576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:54.969895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0bee20-dce8-4213-a011-6f80fb55bf2c service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:54.989453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 990/1965] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:54 2022] POST /identity/v3/auth/tokens => generated 3319 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:54.999518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.000976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.034807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.036136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.036421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.036701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] PATH_INFO: `/v3/projects/c129535452294d998fee4c9e259c9d41` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.038836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c129535452294d998fee4c9e259c9d41)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.041029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.060298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5ced93b-db1d-4be0-84b0-b58c70af812e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.063935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.064588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.064986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.065448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.067750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.093842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c129535452294d998fee4c9e259c9d41', 'request_id': 'req-e1ef570e-7f3f-4ad1-8999-2bba910816d1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:55.101106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c129535452294d998fee4c9e259c9d41', 'request_id': 'req-e1ef570e-7f3f-4ad1-8999-2bba910816d1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:55.102907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=94,keystone:DELETE=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:55.110727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c129535452294d998fee4c9e259c9d41', 'request_id': 'req-e1ef570e-7f3f-4ad1-8999-2bba910816d1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:55.123113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] The token cache is being invalidate because project c129535452294d998fee4c9e259c9d41 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:55.123402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:55.124057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e1ef570e-7f3f-4ad1-8999-2bba910816d1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:55.129583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 976/1966] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:54 2022] DELETE /identity/v3/projects/c129535452294d998fee4c9e259c9d41 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:55.130998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.131289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.132107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.144526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.145987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-534a3d44-2060-4e28-82f6-72847000008d service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.154239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:55.155656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 991/1967] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:54 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:55.155771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.226271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 977/1968] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:55.235154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a9675-8d88-409f-9679-2630863e9577 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.235461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a9675-8d88-409f-9679-2630863e9577 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.235809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-168a9675-8d88-409f-9679-2630863e9577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.257185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-168a9675-8d88-409f-9679-2630863e9577 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:55.258712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-168a9675-8d88-409f-9679-2630863e9577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.268633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 992/1969] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 35 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:55.274644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.286252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-735be08f-c698-4e4c-b7a3-1ada8985a677 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.291517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.292337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.292632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.292909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.293640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:55.293946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.295622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.308806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 978/1970] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:55.314031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a9675-8d88-409f-9679-2630863e9577 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.326933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-168a9675-8d88-409f-9679-2630863e9577 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.333287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.335157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.335668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.336113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.337107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.339480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.367730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 993/1971] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:55 2022] POST /identity/v3/projects => generated 375 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:55.371929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.384478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db5cd1c3-9ad5-47e6-a74e-69e79daad8f8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.477454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.478218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.478573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.478940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.479734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.481722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.502103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 979/1972] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:55 2022] POST /identity/v3/users => generated 288 bytes in 131 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:55.507320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.515493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=2,keystone:SELECT=98 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:55.520462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-845c244d-c6be-48bc-b113-6b30d6837402 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.526707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.527764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.528212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.528489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.529557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.531574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.543898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 994/1973] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:55.548709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.558928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d8d7e0-6b26-4ac0-a8b7-f0e6d6792138 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.563698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.564473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.565305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.565786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.566945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.569234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ceb46077-90c3-4e16-bf1b-9e0fbdf2a600 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.582645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 980/1974] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:55 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:55.586903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.597143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdd9229e-0164-4374-a966-db0e5118594c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.601728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.602614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.602962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.603247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] PATH_INFO: `/v3/projects/75686cdd2e114839b8ecf5a4fc1a2dbe/users/1cc2bcd94bdc447bb522e059edd8d49e/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.618196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] RBAC: Authorizing `identity:create_grant(project_id=75686cdd2e114839b8ecf5a4fc1a2dbe, user_id=1cc2bcd94bdc447bb522e059edd8d49e, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.620866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.634221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 995/1975] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:55 2022] PUT /identity/v3/projects/75686cdd2e114839b8ecf5a4fc1a2dbe/users/1cc2bcd94bdc447bb522e059edd8d49e/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:55.644971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.645435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.645545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.671012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] MFA Rules not processed for user `1cc2bcd94bdc447bb522e059edd8d49e`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:55.672244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.715401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:INSERT=4,keystone:DELETE=1,keystone:SELECT=95 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:55.740892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 981/1976] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:55 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:55.751680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:55.754046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:55.829981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0c2c307-df1f-4344-ab22-a603dbe3483c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:55.836313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:55.836966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:55.837487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:55.837855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:55.852080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:55.854489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f202b8ae-3f19-4f52-adc1-8f9df8c56abe service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:55.865661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 996/1977] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:55 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:56.089655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:56.152693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a3fcd54-1790-4893-be11-45c4a50feea6 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:56.157099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:56.157974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.158261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.158624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] PATH_INFO: `/v3/users/1cc2bcd94bdc447bb522e059edd8d49e` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.159873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] RBAC: Authorizing `identity:delete_user(user_id=1cc2bcd94bdc447bb522e059edd8d49e)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:56.161801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:56.209451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1cc2bcd94bdc447bb522e059edd8d49e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.217273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1cc2bcd94bdc447bb522e059edd8d49e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.218133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1cc2bcd94bdc447bb522e059edd8d49e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.226286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1cc2bcd94bdc447bb522e059edd8d49e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.237813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1cc2bcd94bdc447bb522e059edd8d49e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.243696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 982/1978] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:01:56 2022] DELETE /identity/v3/users/1cc2bcd94bdc447bb522e059edd8d49e => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:56.248221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.248684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.248799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.265887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:56.267115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:56.328982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 997/1979] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:56.338614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:56.340031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:56.398157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-442c4a3a-610f-4617-afbc-7e5bfb2f14df admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:56.401805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:56.402454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.402810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.403066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.413818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:56.415458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819fbc55-658b-455d-b18f-8f304bf8c133 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:56.426988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 983/1980] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:56 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:56.701111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:56.702629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:56.720674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-caf72b51-e18a-4f38-b476-fe5f8ac54fac None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:56.726188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:56.727200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.727555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.727905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] PATH_INFO: `/v3/projects/75686cdd2e114839b8ecf5a4fc1a2dbe` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.729150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] RBAC: Authorizing `identity:delete_project(project_id=75686cdd2e114839b8ecf5a4fc1a2dbe)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:56.731131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:56.778164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75686cdd2e114839b8ecf5a4fc1a2dbe', 'request_id': 'req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.780025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=1,keystone:DELETE=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:56.787145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75686cdd2e114839b8ecf5a4fc1a2dbe', 'request_id': 'req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.793694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75686cdd2e114839b8ecf5a4fc1a2dbe', 'request_id': 'req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.804809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] The token cache is being invalidate because project 75686cdd2e114839b8ecf5a4fc1a2dbe was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:01:56.805094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.805843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b2014277-93e9-4f27-a4fd-1d06ad3aabb3 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:01:56.807498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 998/1981] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:01:56 2022] DELETE /identity/v3/projects/75686cdd2e114839b8ecf5a4fc1a2dbe => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:01:56.812431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.812874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.812979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.820611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:56.833306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:56.834530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:56.894998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 984/1982] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:56.902633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.902984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.903272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.922148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:56.923386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:56.931424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 999/1983] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:56 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:56.935764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:56.946525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afaa0ecc-aefa-4146-91c7-aaad957b99f1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:56.957081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:56.957921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.958214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.958492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.959367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:01:56.959682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:56.961690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:56.979767 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 985/1984] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:01:56 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:01:56.981892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:56.992872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-991f1b6a-e81f-4418-a4bd-5b2a0d59bc66 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:56.997834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:56.998503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:56.998810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:56.999094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:56.999801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.001735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.036093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1000/1985] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:56 2022] POST /identity/v3/projects => generated 375 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:57.037698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.055423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cafa6c1-b276-4c1c-a579-c45eeffc48b6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.119669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.120445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.120807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.121157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.121943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.123901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.148917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 986/1986] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/users => generated 288 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:57.150126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.160420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85264f98-8fb2-4dd6-8dcf-00930f52d67f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.164950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.165604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.165890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.166167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.166923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.168546 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.178926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1001/1987] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:57.182725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.193341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bd56066-bc3d-471b-9ca8-2382e464b244 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.198969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.199572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.199862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.200142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.200868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.202467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39412c03-ff12-4651-b3ad-ff008d2b9c53 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.213145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 987/1988] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:01:57.217702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.229909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c41c104-e916-4593-a070-5c1adc7f5466 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.234656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.235589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.235873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.236149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] PATH_INFO: `/v3/projects/3e381456ff184b55970645064814206c/users/ff6b7eb5cbb849ac9ea32e7c0f97fb5d/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.251936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3e381456ff184b55970645064814206c, user_id=ff6b7eb5cbb849ac9ea32e7c0f97fb5d, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.255209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.268488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1002/1989] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:57 2022] PUT /identity/v3/projects/3e381456ff184b55970645064814206c/users/ff6b7eb5cbb849ac9ea32e7c0f97fb5d/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:57.279928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.280222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.280504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.307479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] MFA Rules not processed for user `ff6b7eb5cbb849ac9ea32e7c0f97fb5d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:57.308718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:57.371413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:57.375365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 988/1990] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:57.379950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.440842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a5f8473-d886-4e1b-91b1-354ac5db71e4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.445821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.446482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.446791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.447066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.447770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.449746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.479434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1003/1991] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/projects => generated 379 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:57.480939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.492104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab15384d-15b8-44c6-8af3-f1be0b5c61d9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.551338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.552080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.552403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.552744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.553510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.555572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.580613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 989/1992] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/users => generated 290 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:57.582078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.592971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3ec0b97-7669-4f69-a6c1-5f0dc5b021be admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.598098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.598760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.599064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.599338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.600088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.601676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.616142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1004/1993] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:57.618028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.631788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b45e7b9-c4f8-4b5c-b120-4aa92ca3b2a8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.637072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.637742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.638023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.638329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.639193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.640843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-864645c7-8dd1-4b6b-b492-37daad06db9d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.651744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 990/1994] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:57 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:57.655939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.666425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28a982a3-2f9f-4f63-9dd8-ab24feca1949 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.671121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.672006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.672283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.672554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] PATH_INFO: `/v3/projects/b321b6e5c13f413ba8a4fefa65941638/users/119e09220d8b439aa424d2bbcfde038b/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.689128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b321b6e5c13f413ba8a4fefa65941638, user_id=119e09220d8b439aa424d2bbcfde038b, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.692421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.714840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1005/1995] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:57 2022] PUT /identity/v3/projects/b321b6e5c13f413ba8a4fefa65941638/users/119e09220d8b439aa424d2bbcfde038b/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:57.723998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.724297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.724582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.734017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.735789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:57.755560 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] MFA Rules not processed for user `119e09220d8b439aa424d2bbcfde038b`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:57.756783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:57.768732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=5,keystone:DELETE=3,keystone:SELECT=92 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:57.801117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abc874e7-6e33-4442-9e18-5e8c06a9af30 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.804816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.805473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.805772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.806037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.807902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:57.825479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 991/1996] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/auth/tokens => generated 3246 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:01:57.830119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.871290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.872786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.890116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1006/1997] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:57 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 157 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:01:57.894355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d7d62b7-0ce7-4c39-927d-8433a3020f6e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:57.899093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:57.899724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:57.900012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:57.900293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:57.900984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:57.902911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:57.916141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:57.929695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 992/1998] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/projects => generated 379 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:01:57.934562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:57.946330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55304928-78d3-42b7-bc89-ec2a87c78456 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.012496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.013247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.013525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.013800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.014511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.016606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.043909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1007/1999] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:01:57 2022] POST /identity/v3/users => generated 296 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:01:58.045343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.058429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e361fbda-cc08-49bb-b02a-6797c1676836 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.063872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.064881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.065351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.065713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.066864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.068736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.082858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 993/2000] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:58 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:58.085554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.103058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bbc15405-c297-4a50-9c33-7ed0b7d679df admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.108893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.110007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.110364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.110864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] PATH_INFO: `/v3/projects/4de236c99fc54be2b2319705cfb8c1f2/users/45959774c0bc441c8cbbfb6322af285c/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.130497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4de236c99fc54be2b2319705cfb8c1f2, user_id=45959774c0bc441c8cbbfb6322af285c, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.133496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.148778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1008/2001] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:58 2022] PUT /identity/v3/projects/4de236c99fc54be2b2319705cfb8c1f2/users/45959774c0bc441c8cbbfb6322af285c/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:58.154300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.173190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-504ecc96-2375-46fa-ace1-e55ea9d2f380 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.250564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.251591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.252062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.252727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.253777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.255681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-96a1568e-0253-41ec-a67d-aee5dc0ba88f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.267503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 994/2002] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:01:58 2022] GET /identity/v3/roles => generated 1420 bytes in 114 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:01:58.272288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.283209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb3529c2-a7e6-46df-aee8-1c9f3a53f545 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.289640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.290302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.290593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.290923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] PATH_INFO: `/v3/projects/4de236c99fc54be2b2319705cfb8c1f2/users/45959774c0bc441c8cbbfb6322af285c/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.292980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4de236c99fc54be2b2319705cfb8c1f2, user_id=45959774c0bc441c8cbbfb6322af285c, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.295718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.310667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1009/2003] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:01:58 2022] PUT /identity/v3/projects/4de236c99fc54be2b2319705cfb8c1f2/users/45959774c0bc441c8cbbfb6322af285c/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:01:58.323364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.323790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.324224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.371083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] MFA Rules not processed for user `45959774c0bc441c8cbbfb6322af285c`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:01:58.372625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:58.480096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 995/2004] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:01:58 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:01:58.498865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.564382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:58.581342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc5df3d5-4a70-40bf-857b-a3d9dba76068 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.585719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.586476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.586694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.587146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.604462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.604462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.616706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1010/2005] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:58 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:58.638984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.640847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:58.653732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5f97521d-25cc-4fac-bbe4-95155c3bb9da None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.659156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.660018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.660380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.660668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.662756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:01:58.663427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:01:58.671872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8af44010-205a-475b-a1ec-620836790b4f service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:01:58.676290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:01:58.676936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:01:58.677225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:01:58.677503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:01:58.728082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.729531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.733509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:INSERT=1,keystone:SELECT=99 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:01:58.736387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:01:58.739114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-15ba0ac5-567d-4aea-aa52-2df37bf36d82 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:01:58.746895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 996/2006] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:58 2022] GET /identity/v3/auth/tokens => generated 3319 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:01:58.748350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1011/2007] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:01:58 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:04.807981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:04.886673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4ff0a738-166b-43ff-b3e1-a48d84ca38fe service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:04.891644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:04.892542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:04.893394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:04.893688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] PATH_INFO: `/v3/users/ff6b7eb5cbb849ac9ea32e7c0f97fb5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:04.895196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ff6b7eb5cbb849ac9ea32e7c0f97fb5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:04.897066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:04.953168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ff6b7eb5cbb849ac9ea32e7c0f97fb5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:04.964804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ff6b7eb5cbb849ac9ea32e7c0f97fb5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:04.965589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ff6b7eb5cbb849ac9ea32e7c0f97fb5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:04.973762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ff6b7eb5cbb849ac9ea32e7c0f97fb5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:04.986892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ff6b7eb5cbb849ac9ea32e7c0f97fb5d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:04.993401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 997/2008] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:04 2022] DELETE /identity/v3/users/ff6b7eb5cbb849ac9ea32e7c0f97fb5d => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:05.001423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:05.002062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:05.002062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:05.020572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:05.021941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:05.101514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1012/2009] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:05.113692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:05.115507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:05.184166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-824a93f0-eb82-4d97-9bb5-6cb08d2c5f05 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:05.187943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:05.188602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:05.188890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:05.189172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:05.200529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:05.202488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:05.215349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 998/2010] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:05 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:05.516791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:05.518421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:05.535605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3989686f-ed71-4cbd-9ed1-ca5826fd81a6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:05.541907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:05.542733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:05.543035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:05.543374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] PATH_INFO: `/v3/projects/3e381456ff184b55970645064814206c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:05.544684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] RBAC: Authorizing `identity:delete_project(project_id=3e381456ff184b55970645064814206c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:05.546629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:05.594715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3e381456ff184b55970645064814206c', 'request_id': 'req-38502d35-73b7-40f5-81f0-eb1f18c2acbe'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.602391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3e381456ff184b55970645064814206c', 'request_id': 'req-38502d35-73b7-40f5-81f0-eb1f18c2acbe'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.608126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3e381456ff184b55970645064814206c', 'request_id': 'req-38502d35-73b7-40f5-81f0-eb1f18c2acbe'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.609927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:05.620946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] The token cache is being invalidate because project 3e381456ff184b55970645064814206c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:05.621272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.621921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.623669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1013/2011] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:05 2022] DELETE /identity/v3/projects/3e381456ff184b55970645064814206c => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:05.628179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:05.638143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c3229920-9cae-470f-93af-403b28be6369 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:05.655347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:05.698555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:05.699472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:05.699758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:05.700036 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] PATH_INFO: `/v3/users/119e09220d8b439aa424d2bbcfde038b` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:05.701218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=119e09220d8b439aa424d2bbcfde038b)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:05.703181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:05.751090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '119e09220d8b439aa424d2bbcfde038b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.758937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '119e09220d8b439aa424d2bbcfde038b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.759733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '119e09220d8b439aa424d2bbcfde038b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.767563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '119e09220d8b439aa424d2bbcfde038b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.783151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fae59d52-01b2-4d84-9445-ba8abbba4ff5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '119e09220d8b439aa424d2bbcfde038b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:05.789809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 999/2012] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:05 2022] DELETE /identity/v3/users/119e09220d8b439aa424d2bbcfde038b => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:05.857160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:05.858635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:05.872288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38502d35-73b7-40f5-81f0-eb1f18c2acbe admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:05.938301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:05.939241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:05.940361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:05.940731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] PATH_INFO: `/v3/users/e32e398f4c5b43a28da845a71344183a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:05.942007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] RBAC: Authorizing `identity:delete_user(user_id=e32e398f4c5b43a28da845a71344183a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:05.944400 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:05.995053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e32e398f4c5b43a28da845a71344183a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.003466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e32e398f4c5b43a28da845a71344183a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.004355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e32e398f4c5b43a28da845a71344183a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.011656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e32e398f4c5b43a28da845a71344183a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.012355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e32e398f4c5b43a28da845a71344183a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.029978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1014/2013] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:05 2022] DELETE /identity/v3/users/e32e398f4c5b43a28da845a71344183a => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:06.035413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.036254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.036543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.057136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:06.058365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.088342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.089699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.123404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1000/2014] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:06.135746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.137440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.165442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-641179dc-3991-4e44-8af6-bd3b444e090f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:06.170284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:06.171202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.171483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.172314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] PATH_INFO: `/v3/projects/b321b6e5c13f413ba8a4fefa65941638` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.179711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] RBAC: Authorizing `identity:delete_project(project_id=b321b6e5c13f413ba8a4fefa65941638)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:06.182067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:06.182554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:06.204700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66d8e743-f2d4-4508-b942-5b51effad6b5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:06.209202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:06.209347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:06.210132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.210437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.210717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.221749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:06.223426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:06.234982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1001/2015] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:06.248394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b321b6e5c13f413ba8a4fefa65941638', 'request_id': 'req-606dacdf-50df-42ff-ae95-28aab2e8d79d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.257166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b321b6e5c13f413ba8a4fefa65941638', 'request_id': 'req-606dacdf-50df-42ff-ae95-28aab2e8d79d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.263135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b321b6e5c13f413ba8a4fefa65941638', 'request_id': 'req-606dacdf-50df-42ff-ae95-28aab2e8d79d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.277457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] The token cache is being invalidate because project b321b6e5c13f413ba8a4fefa65941638 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:06.277758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.279438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.281611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1015/2016] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:06 2022] DELETE /identity/v3/projects/b321b6e5c13f413ba8a4fefa65941638 => generated 0 bytes in 194 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:06.286180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.296657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afafd19b-d0dd-4d2d-a577-73ac567bc4b2 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:06.359858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:06.360745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.361096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.361933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] PATH_INFO: `/v3/users/45959774c0bc441c8cbbfb6322af285c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.363309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] RBAC: Authorizing `identity:delete_user(user_id=45959774c0bc441c8cbbfb6322af285c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:06.365216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:06.432186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '45959774c0bc441c8cbbfb6322af285c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.450963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '45959774c0bc441c8cbbfb6322af285c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.451817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '45959774c0bc441c8cbbfb6322af285c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.464610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '45959774c0bc441c8cbbfb6322af285c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.484045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '45959774c0bc441c8cbbfb6322af285c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.492302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1002/2017] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:06 2022] DELETE /identity/v3/users/45959774c0bc441c8cbbfb6322af285c => generated 0 bytes in 206 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:06.652787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.652787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.681396 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-606dacdf-50df-42ff-ae95-28aab2e8d79d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:06.752341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:06.753233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.753522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.753796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] PATH_INFO: `/v3/projects/77db5d77e9f74d8980c9516e732f5855` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.755047 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] RBAC: Authorizing `identity:delete_project(project_id=77db5d77e9f74d8980c9516e732f5855)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:06.756982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:06.812880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '77db5d77e9f74d8980c9516e732f5855', 'request_id': 'req-6fc14be5-20a1-4083-b341-892f4298a400'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.822419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '77db5d77e9f74d8980c9516e732f5855', 'request_id': 'req-6fc14be5-20a1-4083-b341-892f4298a400'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.829537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '77db5d77e9f74d8980c9516e732f5855', 'request_id': 'req-6fc14be5-20a1-4083-b341-892f4298a400'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.841995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] The token cache is being invalidate because project 77db5d77e9f74d8980c9516e732f5855 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:06.842292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.842986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:06.844772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1016/2018] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:06 2022] DELETE /identity/v3/projects/77db5d77e9f74d8980c9516e732f5855 => generated 0 bytes in 196 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:06.848995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.850425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.863463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-feeaeafc-9953-4b4a-97d0-3a44ffed653d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:06.902265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:06.904015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:06.985821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:06.987296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:06.987296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:06.987729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] PATH_INFO: `/v3/users/70f874188d9a4bbbba8be4074840aed8` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:06.990595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=70f874188d9a4bbbba8be4074840aed8)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:06.993234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:06.994986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:06.996876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:07.041548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fc14be5-20a1-4083-b341-892f4298a400 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.048538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.049600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.049886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.050153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] PATH_INFO: `/v3/projects/4de236c99fc54be2b2319705cfb8c1f2` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.053689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4de236c99fc54be2b2319705cfb8c1f2)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.055785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.084811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '70f874188d9a4bbbba8be4074840aed8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.093800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '70f874188d9a4bbbba8be4074840aed8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.094692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '70f874188d9a4bbbba8be4074840aed8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.110902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '70f874188d9a4bbbba8be4074840aed8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.126438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a5f7e7fc-94e3-41cf-873a-a50807f441a3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '70f874188d9a4bbbba8be4074840aed8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.134897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4de236c99fc54be2b2319705cfb8c1f2', 'request_id': 'req-ee061396-02a0-4cb6-bf14-c3e6f812cb97'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.135101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1003/2019] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:06 2022] DELETE /identity/v3/users/70f874188d9a4bbbba8be4074840aed8 => generated 0 bytes in 287 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:07.144854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4de236c99fc54be2b2319705cfb8c1f2', 'request_id': 'req-ee061396-02a0-4cb6-bf14-c3e6f812cb97'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.152752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4de236c99fc54be2b2319705cfb8c1f2', 'request_id': 'req-ee061396-02a0-4cb6-bf14-c3e6f812cb97'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.165791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] The token cache is being invalidate because project 4de236c99fc54be2b2319705cfb8c1f2 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:07.166075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.166893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ee061396-02a0-4cb6-bf14-c3e6f812cb97 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.169327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1017/2020] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:06 2022] DELETE /identity/v3/projects/4de236c99fc54be2b2319705cfb8c1f2 => generated 0 bytes in 268 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:07.177401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.177696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.178378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.206051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:07.207302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.304060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1004/2021] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:07.320520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.320812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.321093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.345957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:07.347523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.357671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1018/2022] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 39 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:07.362637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.373808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ce4bc2b-d83f-47bf-a783-2ff192b0d3cf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.382730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.383580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.383967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.384315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.385120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:07.385490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.387211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.403182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1005/2023] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 42 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:02:07.407735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.420911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bca1d3d8-f2f7-4722-92e1-804ccd8b0bd9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.428453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.429533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.430011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.430480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.431756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.435984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.467411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1019/2024] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/projects => generated 381 bytes in 57 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:07.467571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.467837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.468313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.485822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58241c9e-91cf-434a-aac1-b6418df31eaa admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.537362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-069d2d26-1a91-4270-be4b-5be39d0c491d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.542523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.543286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.543571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.543865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.544556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.546552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.546663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.547491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.547776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.548055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] PATH_INFO: `/v3/projects/f2a3553cb2c04be583bb38753177a2b1` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.549794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f2a3553cb2c04be583bb38753177a2b1)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.551968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.565679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:07.573251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1020/2025] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/users => generated 291 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:07.574144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.584613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e3a0170-0d76-4465-87e9-56ce62d01ff9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.590022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.590697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.591015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.591326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.592096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.593940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.608150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1021/2026] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:07.609027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.620253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-19281b5f-d116-4d20-ac76-04f6d0bcd12a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.624214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f2a3553cb2c04be583bb38753177a2b1', 'request_id': 'req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.631912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f2a3553cb2c04be583bb38753177a2b1', 'request_id': 'req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.638461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:DELETE=7,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:07.640980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f2a3553cb2c04be583bb38753177a2b1', 'request_id': 'req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.661633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] The token cache is being invalidate because project f2a3553cb2c04be583bb38753177a2b1 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:07.661931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.662640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.668336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1006/2027] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:07 2022] DELETE /identity/v3/projects/f2a3553cb2c04be583bb38753177a2b1 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:07.672773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.675450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.715675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.715675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.715675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.715675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.716161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.716161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.725653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1022/2028] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:07 2022] GET /identity/v3/roles => generated 1420 bytes in 117 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:07.729774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.731517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.744926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e25f3494-9b68-4f63-b6f1-bcaed0f94267 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.750061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.751100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.751421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.751726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] PATH_INFO: `/v3/projects/4b9a0a4a64a24d61accea8e86410aefb/users/559541d044644807af20bfce9583e0ea/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.751861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f873ab14-7b61-4d4e-8cfc-f3b8607521f4 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.756634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.757578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.757872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.758152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] PATH_INFO: `/v3/users/8230cacc6a894c2e8d37be25d80e930d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.759405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8230cacc6a894c2e8d37be25d80e930d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.761428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.768200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=4b9a0a4a64a24d61accea8e86410aefb, user_id=559541d044644807af20bfce9583e0ea, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.771061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bde314e4-6181-47ec-93f1-b40ca4b9ffb8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:07.785328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1023/2029] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:07 2022] PUT /identity/v3/projects/4b9a0a4a64a24d61accea8e86410aefb/users/559541d044644807af20bfce9583e0ea/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:07.798125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.798415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.798756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.810801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8230cacc6a894c2e8d37be25d80e930d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.819697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8230cacc6a894c2e8d37be25d80e930d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.820462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8230cacc6a894c2e8d37be25d80e930d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.826319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] MFA Rules not processed for user `559541d044644807af20bfce9583e0ea`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:07.827576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.828240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8230cacc6a894c2e8d37be25d80e930d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.841505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8230cacc6a894c2e8d37be25d80e930d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:07.849286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1007/2030] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:07 2022] DELETE /identity/v3/users/8230cacc6a894c2e8d37be25d80e930d => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:07.897769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1024/2031] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:07.902689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:07.904081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:07.925671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:07.976544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-47775fda-919e-437b-8954-1be97e37d5b4 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:07.981259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:07.981935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:07.982222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:07.982501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:07.983241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:07.985171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.013507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1008/2032] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:07 2022] POST /identity/v3/projects => generated 381 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:08.018069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.084300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e24915d-4335-4bb1-b798-c8cdd6b78952 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.105005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:08.130706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.133223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:08.161144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.161833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.162118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.162417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.163128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.165176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.187539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1025/2033] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:08 2022] POST /identity/v3/users => generated 297 bytes in 171 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:08.191630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.201866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4878aecf-3a75-4634-b86a-6c831a9b48e4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.208215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.208891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.209178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.209457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.210202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.210570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0593da33-e75d-4674-a680-3dfd8c5535f7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.211864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.215865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.216739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.217028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.217338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] PATH_INFO: `/v3/projects/72ba336f4ac545c6be7e9db47c4d7038` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.219170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] RBAC: Authorizing `identity:delete_project(project_id=72ba336f4ac545c6be7e9db47c4d7038)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.221800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.223758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1026/2034] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:08.228144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.238921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e81abcb-efd8-4fa2-a7bf-0d9aa6d0844c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.243964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.244846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.245129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.245403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] PATH_INFO: `/v3/projects/9f5c5a435f6c47ec98d89f197dba9334/users/7876b5a6d0364cb3971adc37850984ee/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.262787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f5c5a435f6c47ec98d89f197dba9334, user_id=7876b5a6d0364cb3971adc37850984ee, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.265685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.275237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '72ba336f4ac545c6be7e9db47c4d7038', 'request_id': 'req-0c9e435d-596f-4d56-84e0-5b30ab998695'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:08.280874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '72ba336f4ac545c6be7e9db47c4d7038', 'request_id': 'req-0c9e435d-596f-4d56-84e0-5b30ab998695'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:08.284010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1027/2035] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:08 2022] PUT /identity/v3/projects/9f5c5a435f6c47ec98d89f197dba9334/users/7876b5a6d0364cb3971adc37850984ee/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:08.285468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.290540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '72ba336f4ac545c6be7e9db47c4d7038', 'request_id': 'req-0c9e435d-596f-4d56-84e0-5b30ab998695'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:08.302846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] The token cache is being invalidate because project 72ba336f4ac545c6be7e9db47c4d7038 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:08.303529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:08.305258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:08.307307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1009/2036] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:08 2022] DELETE /identity/v3/projects/72ba336f4ac545c6be7e9db47c4d7038 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:08.364082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0684b329-c51d-474d-b3d1-6da95e396caa admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.381021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.383437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.383822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.384167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.385006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.387063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cee1543c-21f7-404f-af36-acb57fa00e0d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.403423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1028/2037] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:08 2022] GET /identity/v3/roles => generated 1420 bytes in 118 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:08.407460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.410058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:08.467587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:08.511870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c9e435d-596f-4d56-84e0-5b30ab998695 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.517187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.518948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.519333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.519714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] PATH_INFO: `/v3/projects/9f5c5a435f6c47ec98d89f197dba9334/users/7876b5a6d0364cb3971adc37850984ee/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.522805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f5c5a435f6c47ec98d89f197dba9334, user_id=7876b5a6d0364cb3971adc37850984ee, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.525832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.538708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1010/2038] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:08 2022] PUT /identity/v3/projects/9f5c5a435f6c47ec98d89f197dba9334/users/7876b5a6d0364cb3971adc37850984ee/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 132 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:08.551210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.551523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.551843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.588008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] MFA Rules not processed for user `7876b5a6d0364cb3971adc37850984ee`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:08.589421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:08.664066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:08.674197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1029/2039] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:08 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 125 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:08.685485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:08.687360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:08.773007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba832287-6fa2-4929-882a-5ba6ac348773 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:08.777377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:08.778367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:08.779004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:08.779290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:08.790382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:08.792346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c2ce0d04-1a5a-4c3a-a2ab-6afafa361817 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:08.803162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1011/2040] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:08 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 118 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:09.176457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:09.247796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb701d0a-c4aa-445b-b2e6-403ec8dc6b81 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:09.253139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:09.254044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:09.254326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:09.254604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] PATH_INFO: `/v3/users/559541d044644807af20bfce9583e0ea` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:09.255830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] RBAC: Authorizing `identity:delete_user(user_id=559541d044644807af20bfce9583e0ea)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:09.257901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:09.307742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '559541d044644807af20bfce9583e0ea'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.315602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '559541d044644807af20bfce9583e0ea'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.316229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '559541d044644807af20bfce9583e0ea'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.323607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '559541d044644807af20bfce9583e0ea'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.324237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '559541d044644807af20bfce9583e0ea'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.341815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1030/2041] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:09 2022] DELETE /identity/v3/users/559541d044644807af20bfce9583e0ea => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:09.346429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:09.346935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:09.347029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:09.366066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:09.367332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:09.427367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1012/2042] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:09.436966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:09.438353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:09.497974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-65a5a274-e796-4022-aaf2-0c26498408ee admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:09.501554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:09.502971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:09.503250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:09.503522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:09.512343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:09.513771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:09.523123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1031/2043] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:09.812048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:09.813659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:09.831190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e19cb80f-5ab3-4bfd-b4b8-d3b2dda372f2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:09.836673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:09.837574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:09.837861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:09.838137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] PATH_INFO: `/v3/projects/4b9a0a4a64a24d61accea8e86410aefb` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:09.839644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4b9a0a4a64a24d61accea8e86410aefb)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:09.842148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:09.895968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:09.897482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:09.902974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4b9a0a4a64a24d61accea8e86410aefb', 'request_id': 'req-d5f0195d-5254-41e9-93a4-f4a76d297fe5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.910177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4b9a0a4a64a24d61accea8e86410aefb', 'request_id': 'req-d5f0195d-5254-41e9-93a4-f4a76d297fe5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.911552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36201ac5-f5d3-4f79-89cf-a0e588bcbdaf service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:09.919208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4b9a0a4a64a24d61accea8e86410aefb', 'request_id': 'req-d5f0195d-5254-41e9-93a4-f4a76d297fe5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.920555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:09.935607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:09.948098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] The token cache is being invalidate because project 4b9a0a4a64a24d61accea8e86410aefb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:09.948403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.949191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:09.951738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1013/2044] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:09 2022] DELETE /identity/v3/projects/4b9a0a4a64a24d61accea8e86410aefb => generated 0 bytes in 140 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:09.956502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:09.957884 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:09.994806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:09.995689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:09.996539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:09.996824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] PATH_INFO: `/v3/users/75c780dab9e2416aabc0ce84829308b8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:09.998031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] RBAC: Authorizing `identity:delete_user(user_id=75c780dab9e2416aabc0ce84829308b8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:09.999987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.037003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5f0195d-5254-41e9-93a4-f4a76d297fe5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.042399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.043493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.043803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.044128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] PATH_INFO: `/v3/users/7876b5a6d0364cb3971adc37850984ee` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.045342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7876b5a6d0364cb3971adc37850984ee)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.048103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.059066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '75c780dab9e2416aabc0ce84829308b8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.068481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '75c780dab9e2416aabc0ce84829308b8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.069543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '75c780dab9e2416aabc0ce84829308b8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.077812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '75c780dab9e2416aabc0ce84829308b8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.078713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9cbf748e-8aa4-4e80-a129-b5ffa5112510 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '75c780dab9e2416aabc0ce84829308b8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.097994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1032/2045] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:09 2022] DELETE /identity/v3/users/75c780dab9e2416aabc0ce84829308b8 => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:10.102560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.103082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.103388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.103783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7876b5a6d0364cb3971adc37850984ee'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.115156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7876b5a6d0364cb3971adc37850984ee'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.115868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7876b5a6d0364cb3971adc37850984ee'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.123803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7876b5a6d0364cb3971adc37850984ee'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.127535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:10.128941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.137293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7876b5a6d0364cb3971adc37850984ee'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.144545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1014/2046] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:09 2022] DELETE /identity/v3/users/7876b5a6d0364cb3971adc37850984ee => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:10.198357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1033/2047] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:10.209245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.210751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.277294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76d6b1a3-3906-474f-a344-e4c180852416 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.281260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.281917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.282198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.282477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.313377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:10.358725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.360219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f05ae7a2-b351-429c-ac37-dba85e44dc0d service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.372592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1015/2048] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:10 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 164 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:10.474783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.476361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.492340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2d21314-d977-4f8d-9a60-dab82e7ea307 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.497462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.498272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.498555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.498860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] PATH_INFO: `/v3/projects/9f5c5a435f6c47ec98d89f197dba9334` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.500059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f5c5a435f6c47ec98d89f197dba9334)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.503329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.555386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=86,keystone:DELETE=13,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:10.572029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f5c5a435f6c47ec98d89f197dba9334', 'request_id': 'req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.581389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f5c5a435f6c47ec98d89f197dba9334', 'request_id': 'req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.587045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f5c5a435f6c47ec98d89f197dba9334', 'request_id': 'req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.598848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] The token cache is being invalidate because project 9f5c5a435f6c47ec98d89f197dba9334 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:10.599224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.599972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.601629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1034/2049] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:10 2022] DELETE /identity/v3/projects/9f5c5a435f6c47ec98d89f197dba9334 => generated 0 bytes in 127 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:10.606589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d408e3e-204e-4a43-bea7-1dcc587b836a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.606917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d408e3e-204e-4a43-bea7-1dcc587b836a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.607204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d408e3e-204e-4a43-bea7-1dcc587b836a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.629975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1d408e3e-204e-4a43-bea7-1dcc587b836a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:10.629975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1d408e3e-204e-4a43-bea7-1dcc587b836a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.694484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1016/2050] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:10.699275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.700733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.702405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.703196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.703621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.715478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3076f333-ba6e-4ab2-8b29-184d2f8f4f0c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.721698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.722696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.723261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.723628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] PATH_INFO: `/v3/projects/ee2ddfe60895421e940fdae99604890f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.724465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:10.724839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee2ddfe60895421e940fdae99604890f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.726085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.726969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.737649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1017/2051] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 36 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:10.742208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.751980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3939cfa0-ebf5-46d5-977d-0cb5d1adf203 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.756998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.757667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.757948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.758246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.759007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:10.759311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.760941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.772993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1018/2052] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:10 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:02:10.777269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.782806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee2ddfe60895421e940fdae99604890f', 'request_id': 'req-333fb3c7-74d3-4ec3-ab7b-a39783940c71'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.787315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48797f66-ff1c-4993-8d5f-045c8d66849b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.793985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee2ddfe60895421e940fdae99604890f', 'request_id': 'req-333fb3c7-74d3-4ec3-ab7b-a39783940c71'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.801534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee2ddfe60895421e940fdae99604890f', 'request_id': 'req-333fb3c7-74d3-4ec3-ab7b-a39783940c71'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.817735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] The token cache is being invalidate because project ee2ddfe60895421e940fdae99604890f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:10.818120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.819155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-333fb3c7-74d3-4ec3-ab7b-a39783940c71 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:10.821340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1035/2053] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:10 2022] DELETE /identity/v3/projects/ee2ddfe60895421e940fdae99604890f => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:10.825830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355399a-5c0d-46d8-9b54-968572859bf0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.827090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355399a-5c0d-46d8-9b54-968572859bf0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.827494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3355399a-5c0d-46d8-9b54-968572859bf0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.849169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3355399a-5c0d-46d8-9b54-968572859bf0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:10.850390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3355399a-5c0d-46d8-9b54-968572859bf0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.861815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.862554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.862937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.863217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.863916 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.865818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.896314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1019/2054] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/projects => generated 383 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:10.897368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:10.899125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.915834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1036/2055] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:10.923488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.923980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.924385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.932142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:10.943200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:10.944684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:10.968017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a74fc9ab-1bdc-469c-ac5a-c113a6ef2079 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:10.973553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:10.974220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:10.974505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:10.974910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:10.975485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:10.977396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:10.991403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=87,keystone:DELETE=11,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:11.001261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1020/2056] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:11.006633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.014193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1037/2057] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:11.017386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2216877-95cc-4ad9-bb91-60a57e1dda5d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.018508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.022482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.023180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.023464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.023875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.024608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.026257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.027625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf93e2ce-7f06-40cd-8b8e-e83af771c460 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.032283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.032952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.033304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.033606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.034339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:11.034644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.036342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.039032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1021/2058] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:11.041641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.050065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1038/2059] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:02:11.052516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c343e706-6081-4f33-beb2-1484b1f25869 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.054384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.057194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.057857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.058144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.058420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.059175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.060773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.065121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-10db23ad-3fd7-47df-9396-c1e0a97a27eb admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.070479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.071155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.071438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.071715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.071946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1022/2060] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:11.072392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.074790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.078156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.091117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cba981f7-b565-40dc-a300-54478f5c874d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.096536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.097428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.097818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.098031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] PATH_INFO: `/v3/projects/db0872bdff654de981bc62fa41673262/users/8932307ed7d743609deb205d6cfe455f/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.111704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1039/2061] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/projects => generated 363 bytes in 54 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:11.116278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.117743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=db0872bdff654de981bc62fa41673262, user_id=8932307ed7d743609deb205d6cfe455f, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.120825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1777cc58-c1b9-409d-aa9a-b7f344068f82 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.137678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1023/2062] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:11 2022] PUT /identity/v3/projects/db0872bdff654de981bc62fa41673262/users/8932307ed7d743609deb205d6cfe455f/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:11.149099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.149393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.149667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.180725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] MFA Rules not processed for user `8932307ed7d743609deb205d6cfe455f`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:11.182028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:11.195324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d53cc8bf-89c4-4cc6-bce1-26eb84bd077d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.263130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.263746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.264334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.264334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.264980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.266969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.270290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1024/2063] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 122 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:11.274504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.285833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6073d38-a083-416e-a7e7-179fac974811 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.290802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1040/2064] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/users => generated 282 bytes in 177 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:11.290962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.291330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.291616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.291923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.292609 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.293056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.294593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.304931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a7cc775-9eab-4da4-9c6c-c55924b805ee admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.311405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.312083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.312366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.312647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.313379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.316850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.325992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1025/2065] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/projects => generated 383 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:11.328049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1041/2066] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:11.329686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.332399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.342202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e1fd0dd8-1cb8-4fcf-82f2-cffb7712053c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.374837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:11.384309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:11.404003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e9000ae-b586-44e6-b3a4-8e6d41a1aa4e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.406356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.407091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.407376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.407658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.408441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.409270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.410031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.410137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.410426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.410804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.411568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.413552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.423899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1042/2067] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 92 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:11.428705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.438834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1026/2068] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/users => generated 298 bytes in 109 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:11.443525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.445083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d89b323-19ff-460e-b08f-3a2e22e3533d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.449798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.450699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.451019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.451303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] PATH_INFO: `/v3/projects/cc742357520f4df69c2af2afb772864e/users/e730cf11ca18431798b8b762c0f05659/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.454070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14400a9c-6613-441c-a426-bce654e1d92f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.459662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.460354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.460645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.460927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.461657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.463308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.467443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cc742357520f4df69c2af2afb772864e, user_id=e730cf11ca18431798b8b762c0f05659, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.470121 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91b9f7f7-0552-4c9a-a8b1-e99231abb1a5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.473437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1027/2069] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:11.478332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.484898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1043/2070] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:11 2022] PUT /identity/v3/projects/cc742357520f4df69c2af2afb772864e/users/e730cf11ca18431798b8b762c0f05659/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:11.488919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d08f72cd-8670-4cab-baf1-31aafa555774 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.495871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.496252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.496551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.529844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] MFA Rules not processed for user `e730cf11ca18431798b8b762c0f05659`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:11.531007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:11.550521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.551446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.551734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.552012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] PATH_INFO: `/v3/projects/a4c69816a8004c68bef090953805b279/users/88a7a17fbea54d20aeae44bf2b93e4c8/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.567065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4c69816a8004c68bef090953805b279, user_id=88a7a17fbea54d20aeae44bf2b93e4c8, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.570348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.585246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1028/2071] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:11 2022] PUT /identity/v3/projects/a4c69816a8004c68bef090953805b279/users/88a7a17fbea54d20aeae44bf2b93e4c8/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 107 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:11.589147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.599527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1044/2072] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:11.611765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.613175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:11.652897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-576fcd69-9707-42d4-a45b-56aa3058cf7b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.657671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.658444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.658738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.659056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.659817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.661442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.671904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1029/2073] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/roles => generated 1420 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:11.676606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.680715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20c42c0a-d3f2-4424-ba51-b77d0dbc0856 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.686489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5287d300-f0db-48f2-98b6-c0351b8dae66 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.691828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.693076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.693365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.693655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] PATH_INFO: `/v3/projects/a4c69816a8004c68bef090953805b279/users/88a7a17fbea54d20aeae44bf2b93e4c8/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.695913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4c69816a8004c68bef090953805b279, user_id=88a7a17fbea54d20aeae44bf2b93e4c8, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.698614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-74ea1d33-72b3-4be8-ad15-519c9a1abe97 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.710630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1030/2074] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:11 2022] PUT /identity/v3/projects/a4c69816a8004c68bef090953805b279/users/88a7a17fbea54d20aeae44bf2b93e4c8/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:11.720871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.721162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.721439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.735110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.735827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.736665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.736950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.746864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:11.749991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] MFA Rules not processed for user `88a7a17fbea54d20aeae44bf2b93e4c8`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:11.751187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:11.764094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:11.797791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.799718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.808682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1045/2075] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 198 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:11.828837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1031/2076] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:11 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:11.840240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:11.841699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:11.854993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-521ddb2c-41cf-48f3-974e-ea8b5b7c4bf9 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:11.906336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:11.907015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:11.907298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:11.907625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:11.921402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:11.923101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:11.934493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1046/2077] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:11 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 95 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:12.046663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:12.059733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8febd49-ae5a-4792-bc49-0c08667bbc7a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:12.063959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:12.065452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:12.065956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:12.066424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:12.138579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:12.140279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c82575fa-f3cb-4b80-9753-3fe01622fe7d service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:12.150126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1032/2078] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:12 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:12.416270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:12.481622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c43ae413-4d65-48dc-afba-8800a2b0bed5 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:12.486739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:12.487641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:12.487924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:12.488201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] PATH_INFO: `/v3/users/8932307ed7d743609deb205d6cfe455f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:12.489398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8932307ed7d743609deb205d6cfe455f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:12.491378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:12.539633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8932307ed7d743609deb205d6cfe455f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:12.546540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8932307ed7d743609deb205d6cfe455f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:12.547193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8932307ed7d743609deb205d6cfe455f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:12.549800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:12.556372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8932307ed7d743609deb205d6cfe455f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:12.557476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8932307ed7d743609deb205d6cfe455f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:12.575829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1047/2079] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:12 2022] DELETE /identity/v3/users/8932307ed7d743609deb205d6cfe455f => generated 0 bytes in 160 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:12.581235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:12.581841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:12.581841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:12.599904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:12.601121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:12.660740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1033/2080] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:12.670528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:12.671979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:12.732556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b127a771-a7d8-4751-8084-0be9fc906114 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:12.737139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:12.737837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:12.738118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:12.738394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:12.748109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:12.749750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:12.761283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1048/2081] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:12 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:13.037776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:13.039517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:13.063448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3abf0874-06d0-45ee-9975-b01735f63390 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:13.068438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:13.069577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.069857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.070132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] PATH_INFO: `/v3/projects/db0872bdff654de981bc62fa41673262` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.071506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] RBAC: Authorizing `identity:delete_project(project_id=db0872bdff654de981bc62fa41673262)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:13.073406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:13.117263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:13.121760 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'db0872bdff654de981bc62fa41673262', 'request_id': 'req-57e14de1-feb1-466f-ba40-80eedd4a7bdb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.128497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'db0872bdff654de981bc62fa41673262', 'request_id': 'req-57e14de1-feb1-466f-ba40-80eedd4a7bdb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.135988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'db0872bdff654de981bc62fa41673262', 'request_id': 'req-57e14de1-feb1-466f-ba40-80eedd4a7bdb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.148196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] The token cache is being invalidate because project db0872bdff654de981bc62fa41673262 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:13.148487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.149135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.150926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1034/2082] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:13 2022] DELETE /identity/v3/projects/db0872bdff654de981bc62fa41673262 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:13.154911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:13.165343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-657162b7-6145-44ac-88e6-025b0b3c6561 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:13.227168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:13.228174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.228376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.228697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] PATH_INFO: `/v3/users/88a7a17fbea54d20aeae44bf2b93e4c8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.229972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88a7a17fbea54d20aeae44bf2b93e4c8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:13.232153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:13.293534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88a7a17fbea54d20aeae44bf2b93e4c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.302302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88a7a17fbea54d20aeae44bf2b93e4c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.303116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88a7a17fbea54d20aeae44bf2b93e4c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.310188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88a7a17fbea54d20aeae44bf2b93e4c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.310868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-e4319ac4-c35d-4b38-bf92-a236820f6ed6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88a7a17fbea54d20aeae44bf2b93e4c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.326937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:13.333425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1049/2083] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:13 2022] DELETE /identity/v3/users/88a7a17fbea54d20aeae44bf2b93e4c8 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:13.659526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:13.661010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:13.700508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57e14de1-feb1-466f-ba40-80eedd4a7bdb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:13.761162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:13.762037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.762354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.762635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] PATH_INFO: `/v3/projects/a4c69816a8004c68bef090953805b279` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.763864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4c69816a8004c68bef090953805b279)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:13.765774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:13.813347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4c69816a8004c68bef090953805b279', 'request_id': 'req-6bb349a3-9119-4e2d-840c-9752b5244ced'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.819827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4c69816a8004c68bef090953805b279', 'request_id': 'req-6bb349a3-9119-4e2d-840c-9752b5244ced'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.827974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4c69816a8004c68bef090953805b279', 'request_id': 'req-6bb349a3-9119-4e2d-840c-9752b5244ced'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.839515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] The token cache is being invalidate because project a4c69816a8004c68bef090953805b279 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:13.839812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.840458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6bb349a3-9119-4e2d-840c-9752b5244ced admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:13.842017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1035/2084] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:13 2022] DELETE /identity/v3/projects/a4c69816a8004c68bef090953805b279 => generated 0 bytes in 184 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:13.847007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.847467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.847578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.866088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:13.867330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:13.925563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1050/2085] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:13.933430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.933722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.934000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.953020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:13.954222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:13.962138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1036/2086] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:13.966354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:13.976607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0d779bb-fd48-4220-8b7d-6f61f492f89c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:13.981154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:13.981801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:13.982083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:13.982359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:13.983125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:13.983430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:13.985036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:13.996918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1051/2087] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:02:14.000993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.011203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df86beb9-ef48-466e-a8c3-89cdaf32191e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.015796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.016434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.016734 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.016994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.017887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.019902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.047895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1037/2088] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/projects => generated 383 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:14.050239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.061023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e49a1b33-e07d-43a3-ad6e-14bcd06f5414 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.119023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.119692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.119973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.120247 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.120928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.122897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.146284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1052/2089] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:14.147505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.157769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-da5b0428-6d62-4847-8d77-97aa9e8df925 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.162533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.163254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.163543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.163891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.164619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.166230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.177302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1038/2090] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:14.181160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.192115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e2c4d63-cdea-412b-9c2b-4a82d763829a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.196985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.197631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.197910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.198184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.198935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.200614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8d1a035-3036-41cf-a56d-1c64ae0e2a40 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.211975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1053/2091] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:14.216090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.227891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f5592a7-c96a-4d3f-808a-ab62e56c339d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.232480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.233359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.233642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.233920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] PATH_INFO: `/v3/projects/46e92a209a6946c688cb701140530444/users/0878ccf51bc74eb0a8b0a38c1a7d6f1c/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.249998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=46e92a209a6946c688cb701140530444, user_id=0878ccf51bc74eb0a8b0a38c1a7d6f1c, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.253030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.261050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=85,keystone:DELETE=11,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:14.268808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1039/2092] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:14 2022] PUT /identity/v3/projects/46e92a209a6946c688cb701140530444/users/0878ccf51bc74eb0a8b0a38c1a7d6f1c/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:14.280164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.280456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.280763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.308305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] MFA Rules not processed for user `0878ccf51bc74eb0a8b0a38c1a7d6f1c`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:14.309527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:14.356366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:14.375299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1054/2093] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:14.380139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.442148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b08d18f-93e0-4cf3-bad3-35e84eb755e9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.447205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.447863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.448149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.448427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.449108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.451108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.476269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1040/2094] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/projects => generated 381 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:14.480358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.491644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2b70a54-ccc9-4ca6-8a0c-49f9f3e4807d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.547808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.549049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.549341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.549783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.550511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.552486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.571566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1055/2095] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:14.575896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.587928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94c700b9-25e4-4db7-bb66-8eae36fdd967 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.594098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.594911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.595235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.596142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.596978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.598939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.610071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1041/2096] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:14.614706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.624805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a30ea18-274b-4b4f-b332-b61d8af11a29 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.629669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.630571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.630885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.631162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] PATH_INFO: `/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af/users/1d65400f9743414ba8b76e007c26c640/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.648166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85e45fe2a12d45a9ae4d4fe7f66253af, user_id=1d65400f9743414ba8b76e007c26c640, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.650894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.664313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1056/2097] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:14 2022] PUT /identity/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af/users/1d65400f9743414ba8b76e007c26c640/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:14.668386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.679120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9ee2a7cd-226a-4914-abec-38f06bf60a4a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.740758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.741998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.741998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.741998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.742864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.744609 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d38ab253-c497-47b9-991c-57ea7ac2baf8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.759341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1042/2098] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:14 2022] GET /identity/v3/roles => generated 1420 bytes in 88 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:14.760475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.772792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b81a836b-5209-4be2-a859-e2f7c9b0c427 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:14.778072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:14.779206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.779668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.780120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] PATH_INFO: `/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af/users/1d65400f9743414ba8b76e007c26c640/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.782336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=85e45fe2a12d45a9ae4d4fe7f66253af, user_id=1d65400f9743414ba8b76e007c26c640, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:14.786418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:14.800170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1057/2099] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:14 2022] PUT /identity/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af/users/1d65400f9743414ba8b76e007c26c640/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 40 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:14.810669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:14.811046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:14.811331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:14.849695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] MFA Rules not processed for user `1d65400f9743414ba8b76e007c26c640`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:14.851034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:14.926262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:14.937518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1043/2100] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:14 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 128 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:14.948680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:14.950399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:15.007872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-07c0604d-c522-4b28-b684-18b8b23f41b8 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:15.011379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:15.012008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:15.012293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:15.012564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:15.067371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:15.068805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:15.072880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:15.083959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1058/2101] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:14 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:19.245347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:19.246738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:19.259351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-60cdc5f6-c398-464e-a630-8ccb3620a52f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:19.262960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:19.263593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:19.263881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:19.264163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:19.273345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:19.274783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3d4f5fdd-5a87-4aec-b886-5f76085667ce service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:19.284251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1044/2102] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:19 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 39 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:25.084704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:26.325669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:26.389299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-240b6a58-9f2a-459f-94c7-8cac252e5666 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:26.394832 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:26.395712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:26.395998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:26.396277 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] PATH_INFO: `/v3/users/0878ccf51bc74eb0a8b0a38c1a7d6f1c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:26.397488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] RBAC: Authorizing `identity:delete_user(user_id=0878ccf51bc74eb0a8b0a38c1a7d6f1c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:26.399489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:26.448516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0878ccf51bc74eb0a8b0a38c1a7d6f1c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:26.456225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0878ccf51bc74eb0a8b0a38c1a7d6f1c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:26.456958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0878ccf51bc74eb0a8b0a38c1a7d6f1c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:26.463908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0878ccf51bc74eb0a8b0a38c1a7d6f1c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:26.464542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0878ccf51bc74eb0a8b0a38c1a7d6f1c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:26.486013 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1059/2103] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:26 2022] DELETE /identity/v3/users/0878ccf51bc74eb0a8b0a38c1a7d6f1c => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:26.488490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:26.488954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:26.489045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:26.508205 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:26.509403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:26.581050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1045/2104] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:26.591054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:26.592624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:26.652978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-371a8611-640b-4fbb-a4f5-2fa0b7a5f48f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:26.656688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:26.657332 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:26.657615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:26.657894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:26.668695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:26.670132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:26.680687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1060/2105] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:26.951144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:26.952799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:26.965881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a51b174-536d-404e-916d-2cf830a04386 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:26.971475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:26.972504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:26.972858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:26.973207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] PATH_INFO: `/v3/projects/46e92a209a6946c688cb701140530444` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:26.974488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] RBAC: Authorizing `identity:delete_project(project_id=46e92a209a6946c688cb701140530444)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:26.976721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:27.023716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '46e92a209a6946c688cb701140530444', 'request_id': 'req-2bce2c95-ef95-4230-bcfa-12e61e523f33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.029754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '46e92a209a6946c688cb701140530444', 'request_id': 'req-2bce2c95-ef95-4230-bcfa-12e61e523f33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.037127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '46e92a209a6946c688cb701140530444', 'request_id': 'req-2bce2c95-ef95-4230-bcfa-12e61e523f33'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.048045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] The token cache is being invalidate because project 46e92a209a6946c688cb701140530444 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:27.048337 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.049132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.051347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1046/2106] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:26 2022] DELETE /identity/v3/projects/46e92a209a6946c688cb701140530444 => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:27.055548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:27.065118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-85fc6a1f-03e6-4d88-90aa-34244462a055 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:27.084573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:27.127176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:27.128182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.128465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.128746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] PATH_INFO: `/v3/users/1d65400f9743414ba8b76e007c26c640` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.129971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] RBAC: Authorizing `identity:delete_user(user_id=1d65400f9743414ba8b76e007c26c640)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:27.131985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:27.180339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '1d65400f9743414ba8b76e007c26c640'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.187744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '1d65400f9743414ba8b76e007c26c640'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.188361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '1d65400f9743414ba8b76e007c26c640'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.195650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '1d65400f9743414ba8b76e007c26c640'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.196286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ad2d356c-4994-4dc6-b382-2d9d36e3b383 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '1d65400f9743414ba8b76e007c26c640'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.213084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1061/2107] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:27 2022] DELETE /identity/v3/users/1d65400f9743414ba8b76e007c26c640 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:27.481232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:27.482722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:27.497611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bce2c95-ef95-4230-bcfa-12e61e523f33 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:27.557100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:27.557967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.558256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.558532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] PATH_INFO: `/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.559758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] RBAC: Authorizing `identity:delete_project(project_id=85e45fe2a12d45a9ae4d4fe7f66253af)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:27.561678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:27.568688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:27.631217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '85e45fe2a12d45a9ae4d4fe7f66253af', 'request_id': 'req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.637515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '85e45fe2a12d45a9ae4d4fe7f66253af', 'request_id': 'req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.648703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '85e45fe2a12d45a9ae4d4fe7f66253af', 'request_id': 'req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.663077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] The token cache is being invalidate because project 85e45fe2a12d45a9ae4d4fe7f66253af was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:27.663421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.664108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-c1e32a2e-af3e-45e9-b28f-689576c2ebcb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:27.665707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1047/2108] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:27 2022] DELETE /identity/v3/projects/85e45fe2a12d45a9ae4d4fe7f66253af => generated 0 bytes in 185 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:27.671697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.672557 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.673057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.692573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:27.693801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:27.776417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.776701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1062/2109] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:27.776819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.776989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.801595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:27.802858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:27.814458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1048/2110] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 40 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:27.819489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:27.830008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91b701d2-1a7e-41ce-b0af-a034261fd9ab None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:27.835177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:27.835848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.836136 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.836414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.837150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:27.837458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:27.839094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:27.855159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1063/2111] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:02:27.858060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:27.869524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-241b0fa5-3758-402f-b2ae-2c93a5a5805d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:27.876937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:27.878618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.878969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.879251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.879950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:27.882021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:27.909635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1049/2112] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:27 2022] POST /identity/v3/projects => generated 373 bytes in 52 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:27.913754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:27.924037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-032eabf0-80e7-4a76-915d-687cffafc1c6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:27.950623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:27.988674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:27.989348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:27.989628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:27.989899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:27.990591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:27.992664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.012906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1064/2113] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:27 2022] POST /identity/v3/users => generated 287 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:28.017034 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.027689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2d3cdda-b5b8-48c7-9b30-21cc7e436e6f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.032269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.032925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.033210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.034104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.035269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.036549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.046873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1050/2114] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:28.051672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.062171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-635efad5-95a1-4b24-a124-76e506738de7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.067336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.067999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.068285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.068579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.069559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.071217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bf88a7-a773-4242-8937-4a4dccf46ff7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.082137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1065/2115] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:28.086392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.097251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23633aa2-0677-481e-97b0-da8addc7e96e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.102191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.103130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.103411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.103681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] PATH_INFO: `/v3/projects/b6187c1fb93c4e76b0a4457b3484ea0f/users/ccbbf8da87ea4d73b7739fe96db0364c/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.120702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b6187c1fb93c4e76b0a4457b3484ea0f, user_id=ccbbf8da87ea4d73b7739fe96db0364c, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.123391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.137294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1051/2116] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:28 2022] PUT /identity/v3/projects/b6187c1fb93c4e76b0a4457b3484ea0f/users/ccbbf8da87ea4d73b7739fe96db0364c/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:28.149972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.150259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.150535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.179348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] MFA Rules not processed for user `ccbbf8da87ea4d73b7739fe96db0364c`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:28.180750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:28.251833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1066/2117] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:28 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:28.256621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.318526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77ada553-574e-44af-812d-50776b40ca70 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.323458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.324109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.324396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.324672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.325354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.327290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.353334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1052/2118] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:28 2022] POST /identity/v3/projects => generated 375 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:28.357609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.369036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f75b66c2-f1ad-41fb-bad2-e054c4c4f807 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.427106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.427790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.428075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.428567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.429167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.431146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.453769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1067/2119] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:28 2022] POST /identity/v3/users => generated 294 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:28.458615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.468954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-985509e2-f23f-4644-bfa8-c306192d5c03 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.474460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.475174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.475624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.475905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.476636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.478929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.490932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1053/2120] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:28.495288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.506289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bc8dc5c-9831-4829-b3ff-41f808b50b34 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.510955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.511831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.512138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.512409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] PATH_INFO: `/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64/users/57d4802f09d24bad94939d44e7386269/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.527719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a128008ceb3a4f6fa55f3d045cbedd64, user_id=57d4802f09d24bad94939d44e7386269, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.530413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.543240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1068/2121] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:28 2022] PUT /identity/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64/users/57d4802f09d24bad94939d44e7386269/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:28.547267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.554083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:28.558335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7cfdbe4-dc99-4e51-b0f6-70fa9c3350c5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.619157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.620216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.620512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.620795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.621587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.623181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc78be4b-4beb-43a1-84d0-68f5d36f152f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.637505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1054/2122] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:28 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:28.640948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.649416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:28.655443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcf39cc8-f4b0-4b48-8797-445ed685c4f0 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.663591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.664528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.664813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.665193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] PATH_INFO: `/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64/users/57d4802f09d24bad94939d44e7386269/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.667684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a128008ceb3a4f6fa55f3d045cbedd64, user_id=57d4802f09d24bad94939d44e7386269, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.670915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.682300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1069/2123] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:28 2022] PUT /identity/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64/users/57d4802f09d24bad94939d44e7386269/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:28.693450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.693747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.694328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.724072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] MFA Rules not processed for user `57d4802f09d24bad94939d44e7386269`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:28.725302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:28.794846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1055/2124] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:28 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 102 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:28.806994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:28.811635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:28.875033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92403319-c1d0-48d3-bbd6-2815a9406dd3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:28.879542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:28.880231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:28.880515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:28.880794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:28.939060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:28.941572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ccaf877-dd02-4ed8-a8a9-a5de5617dc28 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:28.951495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1070/2125] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:28 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:31.521250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:31.589288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-995f9060-da35-4d7f-aa04-9c90b71bcad5 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:31.594007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:31.594936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:31.595256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:31.595536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] PATH_INFO: `/v3/users/ccbbf8da87ea4d73b7739fe96db0364c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:31.597626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccbbf8da87ea4d73b7739fe96db0364c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:31.599660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:31.631206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:31.648982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccbbf8da87ea4d73b7739fe96db0364c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:31.656239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccbbf8da87ea4d73b7739fe96db0364c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:31.656852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccbbf8da87ea4d73b7739fe96db0364c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:31.664537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccbbf8da87ea4d73b7739fe96db0364c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:31.677046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccbbf8da87ea4d73b7739fe96db0364c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:31.683031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1056/2126] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:31 2022] DELETE /identity/v3/users/ccbbf8da87ea4d73b7739fe96db0364c => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:31.687552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:31.688041 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:31.688133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:31.704812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:31.706182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:31.763665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1071/2127] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:31 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:31.773262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:31.774669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:31.847358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38dd933c-f090-4b27-9282-be7759e17b3b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:31.851387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:31.852213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:31.852495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:31.852766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:31.862761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:31.864177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:31.876250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1057/2128] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:31 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:32.187645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:32.189238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:32.204195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09b4c653-4b28-4cb1-af36-142f149d433a None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:32.209727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:32.210887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:32.211281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:32.211684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] PATH_INFO: `/v3/projects/b6187c1fb93c4e76b0a4457b3484ea0f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:32.213146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] RBAC: Authorizing `identity:delete_project(project_id=b6187c1fb93c4e76b0a4457b3484ea0f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:32.215442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:32.256613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=1,keystone:DELETE=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:32.265090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b6187c1fb93c4e76b0a4457b3484ea0f', 'request_id': 'req-3c5d81e3-69eb-435f-a266-d877e8c81c89'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.272440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b6187c1fb93c4e76b0a4457b3484ea0f', 'request_id': 'req-3c5d81e3-69eb-435f-a266-d877e8c81c89'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.278044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b6187c1fb93c4e76b0a4457b3484ea0f', 'request_id': 'req-3c5d81e3-69eb-435f-a266-d877e8c81c89'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.289531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] The token cache is being invalidate because project b6187c1fb93c4e76b0a4457b3484ea0f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:32.289758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.290385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.291972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1072/2129] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:32 2022] DELETE /identity/v3/projects/b6187c1fb93c4e76b0a4457b3484ea0f => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:32.296009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:32.306523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a956f94-a2e6-41b0-ba31-0f9b4e4e04e4 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:32.363657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:32.364535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:32.364827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:32.365102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] PATH_INFO: `/v3/users/57d4802f09d24bad94939d44e7386269` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:32.366345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] RBAC: Authorizing `identity:delete_user(user_id=57d4802f09d24bad94939d44e7386269)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:32.368288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:32.419152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '57d4802f09d24bad94939d44e7386269'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.421616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:32.429204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '57d4802f09d24bad94939d44e7386269'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.429906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '57d4802f09d24bad94939d44e7386269'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.437875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '57d4802f09d24bad94939d44e7386269'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.451020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-570cd445-b23d-4f4c-be0e-299406ee1e07 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '57d4802f09d24bad94939d44e7386269'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.457406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1058/2130] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:32 2022] DELETE /identity/v3/users/57d4802f09d24bad94939d44e7386269 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:32.745192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:32.747329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:32.773761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5d81e3-69eb-435f-a266-d877e8c81c89 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:32.838043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:32.838896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:32.839184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:32.839464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] PATH_INFO: `/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:32.840668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a128008ceb3a4f6fa55f3d045cbedd64)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:32.842657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:32.890217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a128008ceb3a4f6fa55f3d045cbedd64', 'request_id': 'req-2afa6393-f86a-4d5b-bd04-648293446da4'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.897349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a128008ceb3a4f6fa55f3d045cbedd64', 'request_id': 'req-2afa6393-f86a-4d5b-bd04-648293446da4'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.903127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a128008ceb3a4f6fa55f3d045cbedd64', 'request_id': 'req-2afa6393-f86a-4d5b-bd04-648293446da4'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.914473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] The token cache is being invalidate because project a128008ceb3a4f6fa55f3d045cbedd64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:32.914797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.915461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2afa6393-f86a-4d5b-bd04-648293446da4 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:32.917199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1073/2131] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:32 2022] DELETE /identity/v3/projects/a128008ceb3a4f6fa55f3d045cbedd64 => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:32.922165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:32.922674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:32.922791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:32.942170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:32.943419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:33.004672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1059/2132] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:32 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:33.012558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.012853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.013135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.032001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:33.033750 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:33.041901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1074/2133] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:33.046186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.056937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4ba0d53-7899-4cf2-858f-e5f05d8d6a7f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.061536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.062197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.062487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.062795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.063525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:33.063838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.065559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.077302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1060/2134] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:02:33.081743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.091829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3d9f3137-4c99-418e-8929-03129bfad861 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.097382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.098090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.098463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.098751 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.099492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.101433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.128193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1075/2135] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:33 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:02:33.132431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.142517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31556b47-2f89-4638-aa61-2ee17a9f0add admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.203618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.204310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.204595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.204873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.205747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.207739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.227699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1061/2136] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:33 2022] POST /identity/v3/users => generated 288 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:33.233061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.245345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62adb1ec-f8b2-49fe-94a4-db9de05318e9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.250116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.250889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.251174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.251446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.252492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.254543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.265932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1076/2137] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:33.269891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.280040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc0c9e3f-63db-4db2-98ac-163c7b15b628 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.285470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.286127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.286445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.286724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.287492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.289260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28707b12-9199-4292-8b24-3f7742d65827 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.300761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1062/2138] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:33 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:33.305072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.315169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5b5d74b-5526-4c58-8770-dc01a621ec25 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.320209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.321133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.321418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.321713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] PATH_INFO: `/v3/projects/e33dfa1646b846c5a98f6d4a0fb4f098/users/6c6ea80a52e44276a19bb88c24606451/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.337842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] RBAC: Authorizing `identity:create_grant(project_id=e33dfa1646b846c5a98f6d4a0fb4f098, user_id=6c6ea80a52e44276a19bb88c24606451, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.340684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.342872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=86,keystone:DELETE=11,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:33.355700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1077/2139] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:33 2022] PUT /identity/v3/projects/e33dfa1646b846c5a98f6d4a0fb4f098/users/6c6ea80a52e44276a19bb88c24606451/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:33.422994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.423292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.423570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.450950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] MFA Rules not processed for user `6c6ea80a52e44276a19bb88c24606451`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:33.452173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:33.479589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=3,keystone:SELECT=94,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:33.537739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1063/2140] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:33 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 116 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:33.548896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:33.550908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:33.613412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35ed84e1-b4e0-4a87-9027-c3a479200290 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:33.620840 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:33.621515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:33.621802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:33.622081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:33.631548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:33.632997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26cd9574-59fd-4007-bd52-1ff5ed4bf2f6 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:33.641787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1078/2141] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:33 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:41.038305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:41.105009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6b2ead7-2c7e-41a5-8d1c-22117ffe3f73 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:41.109658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:41.110530 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.110893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.111195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] PATH_INFO: `/v3/users/6c6ea80a52e44276a19bb88c24606451` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:41.112389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] RBAC: Authorizing `identity:delete_user(user_id=6c6ea80a52e44276a19bb88c24606451)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:41.114323 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:41.162042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6c6ea80a52e44276a19bb88c24606451'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.170139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6c6ea80a52e44276a19bb88c24606451'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.171193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6c6ea80a52e44276a19bb88c24606451'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.179629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6c6ea80a52e44276a19bb88c24606451'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.194996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6c6ea80a52e44276a19bb88c24606451'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.201470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1064/2142] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:41 2022] DELETE /identity/v3/users/6c6ea80a52e44276a19bb88c24606451 => generated 0 bytes in 164 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:41.206090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.206579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.206813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:41.224284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:41.225579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:41.293023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1079/2143] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:41.303395 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:41.304975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:41.355154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:41.370386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14ae57b9-29af-49d2-b2b7-787ce18bd634 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:41.375067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:41.376342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.376342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.376571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:41.388160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:41.389751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09c4cee8-4b6e-4344-b95b-41cbee01907e service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:41.410347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1065/2144] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:41.717130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:41.718881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:41.743120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c537ad2c-e018-43cf-b645-134a0491d951 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:41.765579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:41.766494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.766796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.767098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] PATH_INFO: `/v3/projects/e33dfa1646b846c5a98f6d4a0fb4f098` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:41.769070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] RBAC: Authorizing `identity:delete_project(project_id=e33dfa1646b846c5a98f6d4a0fb4f098)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:41.771219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:41.828689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'e33dfa1646b846c5a98f6d4a0fb4f098', 'request_id': 'req-412b2347-ea63-49ae-8f7a-212b0f65a24f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.837883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'e33dfa1646b846c5a98f6d4a0fb4f098', 'request_id': 'req-412b2347-ea63-49ae-8f7a-212b0f65a24f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.844338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'e33dfa1646b846c5a98f6d4a0fb4f098', 'request_id': 'req-412b2347-ea63-49ae-8f7a-212b0f65a24f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.860135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] The token cache is being invalidate because project e33dfa1646b846c5a98f6d4a0fb4f098 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:41.860135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.860413 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-412b2347-ea63-49ae-8f7a-212b0f65a24f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:41.862384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1080/2145] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:41 2022] DELETE /identity/v3/projects/e33dfa1646b846c5a98f6d4a0fb4f098 => generated 0 bytes in 146 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:41.867622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.867991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.868334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:41.893953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:41.895336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:41.973304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1066/2146] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:41.981664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:41.981954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:41.982240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.004322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:42.005560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:42.014320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1081/2147] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:42.020113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.030272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e01b79dd-526c-4af0-aa9d-277ba1d93775 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.036051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.036749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.037032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.037308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.038035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:02:42.038343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.040007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.060054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1067/2148] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:02:42 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:02:42.064297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.084240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b0b44b4-62a1-44af-91dc-4a8400cf24ec None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.096304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.097213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.097504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.097773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.098473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.100565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.111543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=2,keystone:SELECT=91,keystone:DELETE=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:42.145377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1082/2149] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:02:42 2022] POST /identity/v3/projects => generated 371 bytes in 81 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:42.149551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.160540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0034ded-d948-4d68-aa26-ef65ff702303 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.234446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.235291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.235675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.236215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.237135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.239209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.261507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1068/2150] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:02:42 2022] POST /identity/v3/users => generated 286 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:02:42.265753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.279645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3cab61e-d3a4-41fb-ae9a-2a1435cec288 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.285777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.286453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.286741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.287085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.287827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.289584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.301554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1083/2151] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:02:42.305945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.320692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78a433c8-1f9c-4130-921d-572db1e52f66 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.325800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.326473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.326759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.327079 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.327818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.329607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a5bcd6df-70cd-460f-ac44-2b76f33802a9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.342230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1069/2152] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:02:42 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:02:42.346706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.357844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a6c15ea-3599-40ea-a922-81ac983122fe admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.367861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.368798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.369151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.369640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] PATH_INFO: `/v3/projects/c51d881f79214b688e8f48f606d5c9da/users/94eba1a981154972986e9e4e7d50efde/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.392369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c51d881f79214b688e8f48f606d5c9da, user_id=94eba1a981154972986e9e4e7d50efde, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.395168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.429271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1084/2153] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:02:42 2022] PUT /identity/v3/projects/c51d881f79214b688e8f48f606d5c9da/users/94eba1a981154972986e9e4e7d50efde/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 83 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:02:42.441543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.442366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.442753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.474174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] MFA Rules not processed for user `94eba1a981154972986e9e4e7d50efde`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:42.475440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:42.518893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:42.549754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1070/2154] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:02:42.561305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.563311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:42.624466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a74ed10-c468-4ed3-8e19-89bda27374a9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.630128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.631024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.631310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.631588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.650058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.650058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35c0f9c3-72c9-4814-9306-7735439cf708 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.651984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1085/2155] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:42 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:02:42.678256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:42.768428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e740f629-57e5-4dff-9f19-968fdf216650 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:42.773485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:42.774450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.776335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.776335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] PATH_INFO: `/v3/users/94eba1a981154972986e9e4e7d50efde` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.777148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] RBAC: Authorizing `identity:delete_user(user_id=94eba1a981154972986e9e4e7d50efde)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:42.779275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:42.837271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '94eba1a981154972986e9e4e7d50efde'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:42.845959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '94eba1a981154972986e9e4e7d50efde'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:42.846704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '94eba1a981154972986e9e4e7d50efde'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:42.858626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '94eba1a981154972986e9e4e7d50efde'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:42.872795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b67fb122-0248-4e52-a38d-97c1001bad73 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '94eba1a981154972986e9e4e7d50efde'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:42.880562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1071/2156] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:02:42 2022] DELETE /identity/v3/users/94eba1a981154972986e9e4e7d50efde => generated 0 bytes in 203 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:42.884818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.884973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:42.885183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.885283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:42.885466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.885658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:42.904055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] MFA Rules not processed for user `283a1bfe0e3b45aaa0a33e5fe6b1d85a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:42.905293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:42.909062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:02:42.910257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:42.973346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:42.974072 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1086/2157] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:42.992672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1072/2158] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:02:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:02:43.003842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:43.005466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:43.065839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ea7e537-ad8d-44f6-b6a8-1eb48f9ae577 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:43.069713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:43.070369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:43.070654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:43.070977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:43.081261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:43.082747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:43.093131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1087/2159] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:02:43 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:02:43.540567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:02:43.542613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:02:43.555784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ae1c8db-3b80-4824-9028-71364ef4b331 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:02:43.561782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:02:43.562748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:02:43.563064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:02:43.563340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] PATH_INFO: `/v3/projects/c51d881f79214b688e8f48f606d5c9da` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:02:43.565101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c51d881f79214b688e8f48f606d5c9da)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:02:43.567095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:02:43.631887 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c51d881f79214b688e8f48f606d5c9da', 'request_id': 'req-b55679cb-a2e2-4e17-a57a-618ff76d6947'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:43.641534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c51d881f79214b688e8f48f606d5c9da', 'request_id': 'req-b55679cb-a2e2-4e17-a57a-618ff76d6947'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:43.650156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c51d881f79214b688e8f48f606d5c9da', 'request_id': 'req-b55679cb-a2e2-4e17-a57a-618ff76d6947'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:43.662622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] The token cache is being invalidate because project c51d881f79214b688e8f48f606d5c9da was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:02:43.662960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:43.663743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b55679cb-a2e2-4e17-a57a-618ff76d6947 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:02:43.665769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1073/2160] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:02:43 2022] DELETE /identity/v3/projects/c51d881f79214b688e8f48f606d5c9da => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:02:53.088020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:02:53.659684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=27,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:01.748591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:01.750046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:01.763201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bea31c1-fdeb-4444-8de4-c0b72e26b29f service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:01.823626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:01.824509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:01.824788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:01.825065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] PATH_INFO: `/v3/users/e730cf11ca18431798b8b762c0f05659` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:01.826246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e730cf11ca18431798b8b762c0f05659)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:01.828245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:01.876511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e730cf11ca18431798b8b762c0f05659'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:01.884572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e730cf11ca18431798b8b762c0f05659'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:01.885373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e730cf11ca18431798b8b762c0f05659'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:01.896898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e730cf11ca18431798b8b762c0f05659'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:01.897766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e730cf11ca18431798b8b762c0f05659'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:01.920458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1088/2161] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:03:01 2022] DELETE /identity/v3/users/e730cf11ca18431798b8b762c0f05659 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:01.926028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:01.926574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:01.926574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:01.947722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:01.948969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:02.011373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1074/2162] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:01 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:02.021358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.022834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:02.085220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23a04702-722a-4022-8090-bc99f9f67d13 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.089063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.089829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.090122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.090402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.100084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.101550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1d43037-10ea-4fc6-a380-1b4a6a3b7f0a service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.111637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1089/2163] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:02.406835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.410042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:02.431320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1562ccc9-453c-4e36-b709-9e844ff2f553 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.437324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.438497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.438682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.439008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] PATH_INFO: `/v3/projects/cc742357520f4df69c2af2afb772864e` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.440384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] RBAC: Authorizing `identity:delete_project(project_id=cc742357520f4df69c2af2afb772864e)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.445212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.496942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cc742357520f4df69c2af2afb772864e', 'request_id': 'req-740b6b76-bb3f-47fc-885b-0240bfce0235'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:02.506540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cc742357520f4df69c2af2afb772864e', 'request_id': 'req-740b6b76-bb3f-47fc-885b-0240bfce0235'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:02.520816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cc742357520f4df69c2af2afb772864e', 'request_id': 'req-740b6b76-bb3f-47fc-885b-0240bfce0235'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:02.536414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] The token cache is being invalidate because project cc742357520f4df69c2af2afb772864e was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:03:02.536801 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:02.537823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-740b6b76-bb3f-47fc-885b-0240bfce0235 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:02.540570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1075/2164] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:03:02 2022] DELETE /identity/v3/projects/cc742357520f4df69c2af2afb772864e => generated 0 bytes in 135 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:02.545337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96928b-3c91-4d32-9973-67086809244e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.546167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96928b-3c91-4d32-9973-67086809244e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.546612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a96928b-3c91-4d32-9973-67086809244e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.570154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4a96928b-3c91-4d32-9973-67086809244e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:02.571735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4a96928b-3c91-4d32-9973-67086809244e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:02.589687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:02.635770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1090/2165] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:03:02.643696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a306e31-f785-4248-8294-711d2512c5de None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.644061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a306e31-f785-4248-8294-711d2512c5de None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.644417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a306e31-f785-4248-8294-711d2512c5de None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.662915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5a306e31-f785-4248-8294-711d2512c5de None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:02.664217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5a306e31-f785-4248-8294-711d2512c5de None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:02.676361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1076/2166] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:02.677419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96928b-3c91-4d32-9973-67086809244e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.687535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a96928b-3c91-4d32-9973-67086809244e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.693398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.694310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.694592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.694899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.695630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:03:02.695934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.697545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.710442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1091/2167] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:03:02 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:03:02.714991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a306e31-f785-4248-8294-711d2512c5de None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.725646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a306e31-f785-4248-8294-711d2512c5de None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.731182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.731848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.732118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.732384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.733089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.735112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.766133 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1077/2168] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:03:02 2022] POST /identity/v3/projects => generated 375 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:03:02.767269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.778073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-618de708-fd3a-4af9-913d-897fc0b7ce1d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.841350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.842027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.842309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.842584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.843316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.845257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.866848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1092/2169] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:03:02 2022] POST /identity/v3/users => generated 288 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:03:02.870605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.881718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d77b3d72-354c-4c12-be51-02ae8bab9c14 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.886578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.887223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.887508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.887785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.888515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.890110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.901279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1078/2170] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:02 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:03:02.905261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.917049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87ca5959-3970-4281-a1ad-443ee8853f44 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.924183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.925772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.926079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.926361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.927140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.929253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-68634052-0061-44cc-86b8-576959d06ff1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:02.944787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1093/2171] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:02 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:03:02.948744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:02.960531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5d751ec6-fa6a-4464-a389-ff07a84592a7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:02.966299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:02.967408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:02.967798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:02.968181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] PATH_INFO: `/v3/projects/9f35de83a69243b79dd0b79897c64cf6/users/eb8249ab545b471fb21dead200775140/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:02.990763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] RBAC: Authorizing `identity:create_grant(project_id=9f35de83a69243b79dd0b79897c64cf6, user_id=eb8249ab545b471fb21dead200775140, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:02.993452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:03.007724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1079/2172] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:03:02 2022] PUT /identity/v3/projects/9f35de83a69243b79dd0b79897c64cf6/users/eb8249ab545b471fb21dead200775140/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:03.018999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2adfd-37ce-461c-820d-2372f62a6d73 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:03.019299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2adfd-37ce-461c-820d-2372f62a6d73 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:03.019585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d3b2adfd-37ce-461c-820d-2372f62a6d73 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:03.047195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d3b2adfd-37ce-461c-820d-2372f62a6d73 None None] MFA Rules not processed for user `eb8249ab545b471fb21dead200775140`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:03.048553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d3b2adfd-37ce-461c-820d-2372f62a6d73 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:03.116488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1094/2173] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:03 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:03.127603 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:03.129487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:03.151401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:03.190806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-126cd59f-9213-45ac-9425-4bb130d613ee admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:03.194700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:03.195380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:03.195666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:03.195941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:03.204417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:03.205950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5800176b-b07c-465a-8724-239446ec7ac5 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:03.214910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1080/2174] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:03 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:06.681823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:06.682316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:06.682415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:06.700553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] MFA Rules not processed for user `0e4b9bb864f24c93b2c9547432b8fd96`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:06.701814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:06.713929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:06.768409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1095/2175] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:06 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:06.907524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:06.908044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:06.908145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:06.928667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:06.929868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:06.992363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1081/2176] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:03:06 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:07.046206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:07.047661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:07.126257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce84e953-2ed5-4be1-a3b7-db7f6ba56505 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:07.131675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:07.132336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:07.132622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:07.132907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:07.141749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:07.143930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-809c6ca9-6bfd-4498-bc8e-2f1aae3d1ab6 service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:07.152518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1096/2177] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:03:07 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:03:16.984707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=42 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:17.149670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=49 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:17.487793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:17.595389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-254f9849-949c-41f4-9985-01ffc0b854f2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:17.601775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:17.602668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:17.602984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:17.603265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] PATH_INFO: `/v3/users/eb8249ab545b471fb21dead200775140` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:17.604640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] RBAC: Authorizing `identity:delete_user(user_id=eb8249ab545b471fb21dead200775140)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:17.608483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:17.684399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'eb8249ab545b471fb21dead200775140'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:17.695562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'eb8249ab545b471fb21dead200775140'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:17.696311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'eb8249ab545b471fb21dead200775140'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:17.711608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'eb8249ab545b471fb21dead200775140'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:17.728108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'eb8249ab545b471fb21dead200775140'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:17.739178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1082/2178] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:03:17 2022] DELETE /identity/v3/users/eb8249ab545b471fb21dead200775140 => generated 0 bytes in 253 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:17.743796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:17.744087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:17.744384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:17.772270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:17.773969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:17.873422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1097/2179] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 131 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:17.883768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:17.890162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:17.969936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce569d09-a965-4fe0-bd15-8b252273b95d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:17.975367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:17.976196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:17.976545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:17.977018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:17.988897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:17.990623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1b329904-6eeb-429c-8212-4761e4599d56 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.007955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1083/2180] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 125 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:18.329717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.331553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:18.349607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a51e6e5-4b50-40e5-90c6-3d6ab5150613 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.360742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.365321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.366306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.366809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] PATH_INFO: `/v3/projects/9f35de83a69243b79dd0b79897c64cf6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.370171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] RBAC: Authorizing `identity:delete_project(project_id=9f35de83a69243b79dd0b79897c64cf6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.378824 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.397310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe0e05-b931-4fec-80b3-95d6eff4ac1b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.397310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe0e05-b931-4fec-80b3-95d6eff4ac1b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.397310 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-33fe0e05-b931-4fec-80b3-95d6eff4ac1b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-33fe0e05-b931-4fec-80b3-95d6eff4ac1b None None] MFA Rules not processed for user `8f471531dac249d994f864102f7b9d3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:18.415680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-33fe0e05-b931-4fec-80b3-95d6eff4ac1b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:18.432279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:18.465143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '9f35de83a69243b79dd0b79897c64cf6', 'request_id': 'req-1f7db390-16cd-4a82-8972-5756d33e4755'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:18.476059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '9f35de83a69243b79dd0b79897c64cf6', 'request_id': 'req-1f7db390-16cd-4a82-8972-5756d33e4755'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:18.481527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1084/2181] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:03:18.483618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '9f35de83a69243b79dd0b79897c64cf6', 'request_id': 'req-1f7db390-16cd-4a82-8972-5756d33e4755'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:18.498832 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] The token cache is being invalidate because project 9f35de83a69243b79dd0b79897c64cf6 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:03:18.499202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:18.500631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1f7db390-16cd-4a82-8972-5756d33e4755 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:03:18.502418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1098/2182] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:03:18 2022] DELETE /identity/v3/projects/9f35de83a69243b79dd0b79897c64cf6 => generated 0 bytes in 172 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:18.507292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.507590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.507875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.534906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:18.536109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:18.618806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1085/2183] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:18.622856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.623174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.623451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.642120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:18.643945 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:18.651808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1099/2184] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:18.656025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.666703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f0df4500-a2d4-4331-9c8a-92e4d64d741c None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.671348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.672124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.672416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.672693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.673436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:03:18.673830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.675538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.690802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1086/2185] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:03:18 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:03:18.692306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.701814 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c02d3e53-ec41-49eb-86b6-1acae50a582f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.707112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.707772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.708054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.708355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.709051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.711057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.736127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1100/2186] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/projects => generated 385 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:03:18.740182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.750871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c58ef1c-e45d-4020-b80f-d50428dafefb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.817313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.818497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.818497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.819146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.820040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.822066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.861523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1087/2187] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/users => generated 293 bytes in 122 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:03:18.865235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.876028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d6e550a9-cb5c-48e2-87ba-1149653775f5 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.883194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.883957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.884315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.884686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.885497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.887229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.899573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1101/2188] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:03:18.903657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.914460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90579c9f-7928-45c6-9440-ce4d6867bae0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.919387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.920143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.920498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.920848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.921655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.923439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9a23a84b-c804-4f33-aca7-1ce0bb13e899 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.934262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1088/2189] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:18 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:03:18.938257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:18.950178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b0676e69-c531-4cb8-832f-af6a2b9c3545 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:18.955538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:18.957007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.957291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.957569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] PATH_INFO: `/v3/projects/ee9d5b852be444f48a33b2f4be5c34be/users/ddda2bac031d4635b27421d22bb40fff/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:18.974061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] RBAC: Authorizing `identity:create_grant(project_id=ee9d5b852be444f48a33b2f4be5c34be, user_id=ddda2bac031d4635b27421d22bb40fff, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:18.977149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:18.994347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1102/2190] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:03:18 2022] PUT /identity/v3/projects/ee9d5b852be444f48a33b2f4be5c34be/users/ddda2bac031d4635b27421d22bb40fff/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:18.996058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9921da9-6d6e-4428-8e88-3538d4d95b03 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:18.996562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9921da9-6d6e-4428-8e88-3538d4d95b03 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:18.996645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9921da9-6d6e-4428-8e88-3538d4d95b03 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:19.018156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e9921da9-6d6e-4428-8e88-3538d4d95b03 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:19.019662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e9921da9-6d6e-4428-8e88-3538d4d95b03 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:19.056286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:19.097118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1089/2191] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:18 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:03:19.118022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:19.121146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:19.148828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:19.196801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51547cdd-28e9-46bb-9265-cfc2d5c21bbb admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:19.201276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:19.201276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:19.201800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:19.202552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:19.215949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:19.217518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:19.231652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1103/2192] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:23.703590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:23.704124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:23.704302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:23.774698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] MFA Rules not processed for user `ddda2bac031d4635b27421d22bb40fff`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:23.776327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:23.850265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1090/2193] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:23 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 148 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:23.854856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:23.865906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86ed1569-56b5-4264-a1be-d6eb55a9d241 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:23.873802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:23.874473 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:23.874798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:23.875071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:23.875774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:23.877748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:23.904468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1104/2194] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:03:23 2022] POST /identity/v3/projects => generated 383 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:03:23.909125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:23.920453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17c964b9-b66d-4ece-8ef3-f87e8d31eb94 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:23.978859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:23.979534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:23.979816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:23.980089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:23.980781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:23.982690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:24.005195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1091/2195] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:03:23 2022] POST /identity/v3/users => generated 298 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:03:24.013617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:24.025311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc83653e-406f-46d5-a175-8e49ad1e664e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:24.030269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:24.031006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:24.031297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:24.031580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:24.032322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:24.033948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:24.055005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1105/2196] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:03:24.056829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:24.096521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-08731010-9086-43c6-b130-27f7f7762ed1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:24.102712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:24.103918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:24.113970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:24.113970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] PATH_INFO: `/v3/projects/c0a2ac49b37b459d9c979a47acbe488b/users/af119e9383b844eab16d30caa44ce9db/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:24.144838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0a2ac49b37b459d9c979a47acbe488b, user_id=af119e9383b844eab16d30caa44ce9db, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:24.147564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:24.174120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1092/2197] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:03:24 2022] PUT /identity/v3/projects/c0a2ac49b37b459d9c979a47acbe488b/users/af119e9383b844eab16d30caa44ce9db/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:03:24.178036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:24.188692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe817fc-97e2-4298-9809-943a776f7605 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:24.251344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:24.252756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:24.253045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:24.253320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:24.254057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:24.255748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d714a8aa-4135-4d56-90e3-b79410b9be3e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:24.267027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1106/2198] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:03:24 2022] GET /identity/v3/roles => generated 1420 bytes in 89 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:03:24.271909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:24.283264 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f5a7f9f-2680-4779-83e4-0d96615a51e3 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:24.289931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:24.290874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:24.291161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:24.291443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] PATH_INFO: `/v3/projects/c0a2ac49b37b459d9c979a47acbe488b/users/af119e9383b844eab16d30caa44ce9db/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:24.293447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] RBAC: Authorizing `identity:create_grant(project_id=c0a2ac49b37b459d9c979a47acbe488b, user_id=af119e9383b844eab16d30caa44ce9db, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:24.296712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfe7f648-14a8-4823-8a5d-7348ce898e5a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:24.310796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1093/2199] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:03:24 2022] PUT /identity/v3/projects/c0a2ac49b37b459d9c979a47acbe488b/users/af119e9383b844eab16d30caa44ce9db/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 39 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:03:25.412449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ad583-f939-49ce-ba05-058730b0efea None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:25.413038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ad583-f939-49ce-ba05-058730b0efea None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:25.413147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-415ad583-f939-49ce-ba05-058730b0efea None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:25.414675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1107/2200] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:03:25 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:03:25.424086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248593e-a151-4043-b520-ec2f4bfc0690 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:25.425276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248593e-a151-4043-b520-ec2f4bfc0690 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:25.425732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a248593e-a151-4043-b520-ec2f4bfc0690 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:25.471945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a248593e-a151-4043-b520-ec2f4bfc0690 None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:25.473091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a248593e-a151-4043-b520-ec2f4bfc0690 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:25.513345 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:25.620186 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1094/2201] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 197 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:25.634216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415ad583-f939-49ce-ba05-058730b0efea None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:25.636248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-415ad583-f939-49ce-ba05-058730b0efea None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:25.713798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-415ad583-f939-49ce-ba05-058730b0efea None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:25.718679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:25.719625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:25.719841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:25.720721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:25.758244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:25.759853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-826184e1-e26e-4670-8a9d-0de0618ccfcd service glance] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:25.788241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1108/2202] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:25 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:25.886276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:25.886276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:25.886276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:25.888627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1095/2203] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:03:25 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:03:25.895592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db43358-07de-42bf-a05a-444edd9bf1af None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:25.896158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db43358-07de-42bf-a05a-444edd9bf1af None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:25.896637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8db43358-07de-42bf-a05a-444edd9bf1af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:25.919697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8db43358-07de-42bf-a05a-444edd9bf1af None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:25.921261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8db43358-07de-42bf-a05a-444edd9bf1af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:25.965178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:26.052733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1109/2204] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:03:25 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 158 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:26.101126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:26.103252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:26.135610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18fab347-d6cd-4f5f-bafb-a0358397d8e9 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:26.147100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:26.148043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:26.148399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:26.148751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:26.164600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:26.166248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4b10953a-9886-44b9-a401-acc6c5c6aeff service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:26.192554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1096/2205] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:26 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:26.383604 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816763f4-dc02-4391-b52a-e6145e23c39b None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:26.383897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816763f4-dc02-4391-b52a-e6145e23c39b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:26.384178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-816763f4-dc02-4391-b52a-e6145e23c39b None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:26.386166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1110/2206] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:03:26 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:03:26.399913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58decf4-ba53-4250-b509-e9e99d28bd7a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:26.400209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58decf4-ba53-4250-b509-e9e99d28bd7a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:26.400701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d58decf4-ba53-4250-b509-e9e99d28bd7a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:26.448474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-d58decf4-ba53-4250-b509-e9e99d28bd7a None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:26.449706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d58decf4-ba53-4250-b509-e9e99d28bd7a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:26.479550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1097/2207] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:03:26 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:26.563813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816763f4-dc02-4391-b52a-e6145e23c39b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:26.583439 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-816763f4-dc02-4391-b52a-e6145e23c39b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:26.588596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:26.589454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:26.589906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:26.590499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:26.602981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:26.605242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:26.637860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1111/2208] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:26 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 74 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:31.044556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4df003-57c8-4d81-b6f1-132523f0b321 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:31.045014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4df003-57c8-4d81-b6f1-132523f0b321 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:31.045138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd4df003-57c8-4d81-b6f1-132523f0b321 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:31.088427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-bd4df003-57c8-4d81-b6f1-132523f0b321 None None] MFA Rules not processed for user `af119e9383b844eab16d30caa44ce9db`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:31.089697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bd4df003-57c8-4d81-b6f1-132523f0b321 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:31.187735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1098/2209] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:31 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:31.200898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:31.215460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfcc0936-f6ff-40ac-affc-1852d06fcfd8 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:31.220659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:31.221650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:31.221962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:31.222225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:31.289336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:31.290849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:31.301325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1112/2210] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:31 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 101 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:03:31.519382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088f82-31ea-49bb-8b7e-5736bfab83af None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:31.519682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088f82-31ea-49bb-8b7e-5736bfab83af None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:31.519968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a088f82-31ea-49bb-8b7e-5736bfab83af None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:31.556677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3a088f82-31ea-49bb-8b7e-5736bfab83af None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:31.560027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3a088f82-31ea-49bb-8b7e-5736bfab83af None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:31.670371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1099/2211] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:03:31 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 152 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:31.691367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:31.693393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:31.778504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dbbadfb5-52f5-4c9f-ac69-1fdc8db683ee service glance] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:31.783561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:31.784209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:31.784493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:31.784771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:31.793432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:31.795217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e25b4c3-b506-4027-b24d-32866eeb9358 service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:31.804240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1113/2212] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:03:31 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 113 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:03:41.585263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:41.585880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:41.585880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:41.609396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] MFA Rules not processed for user `ddda2bac031d4635b27421d22bb40fff`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:41.610692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:41.627291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1100/2213] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:03:41 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 43 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:41.800934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=76 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:42.601859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:42.602152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:42.602440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:42.640808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:42.642857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:42.676125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1114/2214] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:03:42 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:42.689898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:42.705522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0f1818a0-1cd7-4075-807e-d732e705bd7f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:42.711902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:42.712714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:42.713252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:42.713631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:42.727086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:42.730084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-77080f23-45b2-47c0-bb3b-4bcb94fbb0c8 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:42.735233 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:42.754959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1101/2215] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:03:42 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 67 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 01:03:52.617938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:52.749026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:03:53.771336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:53.772822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:53.844172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3b3c136-3ec4-435b-ba7d-dc89b4cb8a0e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:53.849622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:53.850283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:53.850571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:53.850879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:53.852858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:53.865346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:53.867012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:53.879238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1115/2216] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:03:53 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:03:57.091979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca39e7-507b-4efe-99fa-ae48a1b66898 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:57.092477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca39e7-507b-4efe-99fa-ae48a1b66898 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:57.092580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-42ca39e7-507b-4efe-99fa-ae48a1b66898 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:57.112588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-42ca39e7-507b-4efe-99fa-ae48a1b66898 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:03:57.113809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-42ca39e7-507b-4efe-99fa-ae48a1b66898 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:03:57.121761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1102/2217] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:03:57 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:03:57.135217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:03:57.148725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f9a034f-868b-4d99-b707-4d21a4625cae service placement] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:03:57.153959 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:03:57.154615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:03:57.154947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:03:57.155597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:03:57.165858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:03:57.167685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:03:57.191974 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1116/2218] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:03:57 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:04:07.106749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:07.187385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=38 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:16.742078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181544ec-f500-4080-a289-da42de6668df None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:16.742567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181544ec-f500-4080-a289-da42de6668df None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:16.742672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-181544ec-f500-4080-a289-da42de6668df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:16.761300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-181544ec-f500-4080-a289-da42de6668df None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:16.762452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-181544ec-f500-4080-a289-da42de6668df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:16.769799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1103/2219] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:04:16 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:04:16.795026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:16.806635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-975f9adc-2440-49ae-90c6-19f17883cbaa service swift] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:16.812809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:16.813544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:16.814033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:16.814449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:16.823834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:16.825442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:16.834354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1117/2220] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:04:16 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 40 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:04:26.755606 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:26.831201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=6 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:32.904716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181544ec-f500-4080-a289-da42de6668df None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:32.906675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-181544ec-f500-4080-a289-da42de6668df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:33.004924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-181544ec-f500-4080-a289-da42de6668df None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.009563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.010363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.010651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.010968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] PATH_INFO: `/v3/users/0e4b9bb864f24c93b2c9547432b8fd96` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.012189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0e4b9bb864f24c93b2c9547432b8fd96)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.014228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.064145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0e4b9bb864f24c93b2c9547432b8fd96'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.073380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0e4b9bb864f24c93b2c9547432b8fd96'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.074359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0e4b9bb864f24c93b2c9547432b8fd96'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.082128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0e4b9bb864f24c93b2c9547432b8fd96'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.094340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0e4b9bb864f24c93b2c9547432b8fd96'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.101068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1104/2221] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:04:32 2022] DELETE /identity/v3/users/0e4b9bb864f24c93b2c9547432b8fd96 => generated 0 bytes in 197 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:04:33.291485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.300883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3495cf6f-7908-4920-b246-2c35175779ec service swift] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.350508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.351231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.351512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.351786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.356361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:33.356809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.358387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:33.428200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.429761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5dc6f4a3-9cc2-489e-a238-b52469aab04b service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.439553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1118/2222] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:04:33 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:04:33.455013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6a0a20a-fd27-45f8-8194-506aac303f55 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.460484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.461356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.461643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.461925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] PATH_INFO: `/v3/projects/4dd26bb2797345d6b531f5ba5db5cbc8` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.465049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] RBAC: Authorizing `identity:delete_project(project_id=4dd26bb2797345d6b531f5ba5db5cbc8)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.465178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.488937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:33.514542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '4dd26bb2797345d6b531f5ba5db5cbc8', 'request_id': 'req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.521334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '4dd26bb2797345d6b531f5ba5db5cbc8', 'request_id': 'req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.529640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '4dd26bb2797345d6b531f5ba5db5cbc8', 'request_id': 'req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.541441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] The token cache is being invalidate because project 4dd26bb2797345d6b531f5ba5db5cbc8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:04:33.541736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.542364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-038d1ea5-b2d5-4826-8f7e-06bb2757fa59 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:33.544172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1105/2223] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:04:33 2022] DELETE /identity/v3/projects/4dd26bb2797345d6b531f5ba5db5cbc8 => generated 0 bytes in 188 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:04:33.548942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.549302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.550220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.569997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:33.571415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:33.632126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1119/2224] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:04:33.639923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.640276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.640623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.660096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:33.661354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:33.669286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1106/2225] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:04:33.673667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.683516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab0964dc-7ecf-47cd-8d99-d6e2842d7152 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.688290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.689068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.689358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.689694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.690419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:04:33.690727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.692514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.704601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1120/2226] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:04:33 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:04:33.709400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.719185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d0356e8-e8f6-4644-a17f-8bfd976c9f39 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.724790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.725456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.725792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.726109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.726869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.728819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.757450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1107/2227] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:04:33 2022] POST /identity/v3/projects => generated 397 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:04:33.758978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.770019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0241b0ab-e478-49cc-899f-91269f241c99 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.806501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:33.831689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.832397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.832678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.832953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.833640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.835617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.860194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1121/2228] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:04:33 2022] POST /identity/v3/users => generated 299 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:04:33.861866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.872638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54fa43ee-7341-4fb1-bd75-544c0204f7e0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.877639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.878286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.878726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.879247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.880001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.881650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.894111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1108/2229] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:04:33.898718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.910037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abd13551-2251-4853-8d43-a9273dae8dbe admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.914986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.915688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.915981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.916259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.916996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.918611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7bcbc38a-a38e-4e0a-9db9-199cd4a8344a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.931164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1122/2230] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:04:33 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:04:33.935219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:33.946590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac6126aa-a851-45c9-ae64-ec42b1d553f1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:33.951278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:33.952232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.952583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.952868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] PATH_INFO: `/v3/projects/73cb3e8939cf433fab1647cfd13e3d8a/users/13aad7a216ac42bea7eba7e9cf7606cf/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:33.967966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] RBAC: Authorizing `identity:create_grant(project_id=73cb3e8939cf433fab1647cfd13e3d8a, user_id=13aad7a216ac42bea7eba7e9cf7606cf, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:33.971121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:33.984534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1109/2231] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:04:33 2022] PUT /identity/v3/projects/73cb3e8939cf433fab1647cfd13e3d8a/users/13aad7a216ac42bea7eba7e9cf7606cf/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:04:33.994521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:33.994932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:33.996621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:34.028820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] MFA Rules not processed for user `13aad7a216ac42bea7eba7e9cf7606cf`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:34.030075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:34.099657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1123/2232] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:33 2022] POST /identity/v3/auth/tokens => generated 3264 bytes in 106 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:04:34.111771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:34.113728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:34.173003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d99a5b8-6e62-4a49-8827-a96f412ad9fc admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:34.179615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:34.180469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:34.180750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:34.181024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:34.189692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:34.191876 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c7505fcf-5596-4673-a73b-2945b80b575b service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:34.201162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1110/2233] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:04:34 2022] GET /identity/v3/auth/tokens => generated 3264 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:04:42.441839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:42.553968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87b2584c-858e-432a-94a5-a091bc27753c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:42.558551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:42.559505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:42.559788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:42.560064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] PATH_INFO: `/v3/users/13aad7a216ac42bea7eba7e9cf7606cf` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:42.561409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] RBAC: Authorizing `identity:delete_user(user_id=13aad7a216ac42bea7eba7e9cf7606cf)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:42.563451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:42.614941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13aad7a216ac42bea7eba7e9cf7606cf'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:42.616847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:INSERT=3,keystone:DELETE=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:42.625252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13aad7a216ac42bea7eba7e9cf7606cf'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:42.626033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13aad7a216ac42bea7eba7e9cf7606cf'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:42.634734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13aad7a216ac42bea7eba7e9cf7606cf'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:42.635467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13aad7a216ac42bea7eba7e9cf7606cf'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:42.654163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1124/2234] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:04:42 2022] DELETE /identity/v3/users/13aad7a216ac42bea7eba7e9cf7606cf => generated 0 bytes in 213 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:04:42.659684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:42.660171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:42.660262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:42.683299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:42.684502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:42.744387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=7,keystone:SELECT=90,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:42.767261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1111/2235] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:42 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:04:42.780583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:42.782209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:42.850605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92e67863-0c82-46a8-bec2-c9d082a9f739 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:42.855806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:42.857494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:42.857494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:42.857494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:42.868647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:42.870302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-893fc857-ba88-417f-a0c7-1a7021f344bf service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:42.883000 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1125/2236] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:04:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:04:43.181874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:04:43.183492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:43.196762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e79f3d4b-0a05-4f04-9692-1cda91552849 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:04:43.201708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:04:43.202479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:43.202840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:43.203125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] PATH_INFO: `/v3/projects/73cb3e8939cf433fab1647cfd13e3d8a` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:43.204332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] RBAC: Authorizing `identity:delete_project(project_id=73cb3e8939cf433fab1647cfd13e3d8a)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:04:43.206453 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:04:43.279660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '73cb3e8939cf433fab1647cfd13e3d8a', 'request_id': 'req-b454bdf2-6a1b-4787-b200-52d4cdef2806'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:43.286670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '73cb3e8939cf433fab1647cfd13e3d8a', 'request_id': 'req-b454bdf2-6a1b-4787-b200-52d4cdef2806'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:43.298683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '73cb3e8939cf433fab1647cfd13e3d8a', 'request_id': 'req-b454bdf2-6a1b-4787-b200-52d4cdef2806'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:43.315648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] The token cache is being invalidate because project 73cb3e8939cf433fab1647cfd13e3d8a was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:04:43.315648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:43.316443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b454bdf2-6a1b-4787-b200-52d4cdef2806 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:04:43.322834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1112/2237] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:04:43 2022] DELETE /identity/v3/projects/73cb3e8939cf433fab1647cfd13e3d8a => generated 0 bytes in 137 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:04:52.161367 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d2f58-31d3-4b87-b10f-f680213adbdd None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:52.161868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d2f58-31d3-4b87-b10f-f680213adbdd None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:52.161993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-078d2f58-31d3-4b87-b10f-f680213adbdd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:52.179444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-078d2f58-31d3-4b87-b10f-f680213adbdd None None] MFA Rules not processed for user `283a1bfe0e3b45aaa0a33e5fe6b1d85a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:52.180734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-078d2f58-31d3-4b87-b10f-f680213adbdd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:52.233966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1126/2238] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:52 2022] POST /identity/v3/auth/tokens => generated 3258 bytes in 74 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:04:53.309568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=32,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:04:54.660367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ba8c-fb2a-4993-b01c-a9d985dd7866 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:04:54.660971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ba8c-fb2a-4993-b01c-a9d985dd7866 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:04:54.660971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9330ba8c-fb2a-4993-b01c-a9d985dd7866 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:04:54.679309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-9330ba8c-fb2a-4993-b01c-a9d985dd7866 None None] MFA Rules not processed for user `8f471531dac249d994f864102f7b9d3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:04:54.680497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9330ba8c-fb2a-4993-b01c-a9d985dd7866 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:04:54.733812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1113/2239] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:04:54 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:05:02.227117 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=1,keystone:DELETE=3,keystone:SELECT=63 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:04.727317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=25 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:09.089166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:05:09.089735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:05:09.089735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:05:09.108507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] MFA Rules not processed for user `9a9256b6d7e24c54b061673cd17217cd`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:05:09.110166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:09.166354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1127/2240] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:05:09 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:05:19.159589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=25 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:21.300151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:05:21.300666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:05:21.300777 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:05:21.319263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] MFA Rules not processed for user `a59f5944a0db4710a43a6447019864d2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:05:21.320479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:21.374621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1114/2241] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:05:21 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:05:31.367919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=25 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:34.568859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:05:34.570265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:34.625710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44a1051c-0cb2-4b2a-baec-f1555f9eaf2b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:05:34.629416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:05:34.630052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:05:34.630336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:05:34.630611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:05:34.632403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:34.704636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:05:34.706172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:05:34.717042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1128/2242] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:05:34 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:05:38.133295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:05:38.134732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:38.190684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-600ba6e6-e097-400b-9d28-85dfe48ffed1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:05:38.194723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:05:38.195385 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:05:38.195675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:05:38.195958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:05:38.197797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:38.212057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:05:38.213489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:05:38.225105 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1115/2243] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:05:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:05:44.712265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=54 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:48.219537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=26 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:05:55.256613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:05:55.258024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:55.271944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-afb5c5e7-e45f-4f05-b7ee-125a03167dba service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:05:55.277457 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:05:55.278092 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:05:55.278377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:05:55.278656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:05:55.280405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:05:55.291149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:05:55.292735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:05:55.302441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1129/2244] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:05:55 2022] GET /identity/v3/auth/tokens => generated 3258 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:00.709564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:00.718159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11d6c3ad-2730-458a-b438-8d2dfc4d8b4f service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:00.721791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:00.722428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:00.722784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:00.723292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:00.725084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:00.736163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:00.737583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eb0b72f1-5e3b-41dc-a884-6a32b5b1d584 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:00.746522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1116/2245] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:00 2022] GET /identity/v3/auth/tokens => generated 3252 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:01.254436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:01.255917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:01.267290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0b086ead-cb1f-47e8-a5d8-1a6d08d019f7 service nova] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:01.270957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:01.271610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:01.271891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:01.272166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:01.273869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:01.335328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:01.336761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91473153-c549-4d46-999e-87b8317d96ba service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:01.346273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1130/2246] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:01 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:04.692144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:04.692638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:04.692864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:04.694534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1117/2247] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:06:04 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:06:04.702426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:04.702865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:04.703186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:04.721003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] MFA Rules not processed for user `9eaff003925c4f7283d99ad51db217e2`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:04.722217 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:04.799670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1131/2248] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:06:04 2022] POST /identity/v3/auth/tokens => generated 3298 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:04.995039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:04.996857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:05.013468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c1132c0d-b719-4686-baad-99e008a42007 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.018681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.019472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.020489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.020774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.030467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.032232 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.043315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.044494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1118/2249] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:04 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:05.045003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:05.060046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b172462-ea53-4e10-9d1d-0dffe3e7e5a8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.064720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.065598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.065880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.066150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] PATH_INFO: `/v3/users/283a1bfe0e3b45aaa0a33e5fe6b1d85a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.067626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] RBAC: Authorizing `identity:delete_user(user_id=283a1bfe0e3b45aaa0a33e5fe6b1d85a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.069928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.122282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '283a1bfe0e3b45aaa0a33e5fe6b1d85a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.130242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '283a1bfe0e3b45aaa0a33e5fe6b1d85a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.130948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '283a1bfe0e3b45aaa0a33e5fe6b1d85a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.138450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '283a1bfe0e3b45aaa0a33e5fe6b1d85a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.139148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c788217e-6e78-44c4-a183-2c094e5c763a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '283a1bfe0e3b45aaa0a33e5fe6b1d85a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.146961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:05.159552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1132/2250] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:05 2022] DELETE /identity/v3/users/283a1bfe0e3b45aaa0a33e5fe6b1d85a => generated 0 bytes in 118 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:05.337785 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.351362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4b2475f-2bd4-438a-add7-26fa0ec963d7 service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.415280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.416267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.416638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.416990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] PATH_INFO: `/v3/projects/f7d219e82efe44f9b6f860a5fabc7733` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.418266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=f7d219e82efe44f9b6f860a5fabc7733)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.420226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.467281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'f7d219e82efe44f9b6f860a5fabc7733', 'request_id': 'req-1053dca8-8c20-4228-8b7c-3a3460a926d5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.473487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'f7d219e82efe44f9b6f860a5fabc7733', 'request_id': 'req-1053dca8-8c20-4228-8b7c-3a3460a926d5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.480859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'f7d219e82efe44f9b6f860a5fabc7733', 'request_id': 'req-1053dca8-8c20-4228-8b7c-3a3460a926d5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.491772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] The token cache is being invalidate because project f7d219e82efe44f9b6f860a5fabc7733 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:05.492062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.492682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1053dca8-8c20-4228-8b7c-3a3460a926d5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:05.494316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1119/2251] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:05 2022] DELETE /identity/v3/projects/f7d219e82efe44f9b6f860a5fabc7733 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:05.499275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.499564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.499839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.517852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:05.519073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:05.579938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1133/2252] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:05.588300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.588592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.588868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.608546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:05.609743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:05.617970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1120/2253] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:05.622125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.631690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffa711fa-3162-45b4-b936-407566fae7d2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.636164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.636827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.637107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.637383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.638110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:05.638412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.640057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.652486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1134/2254] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:06:05.656978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.666795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-849aadf3-c820-4374-835c-bb70ee800857 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.671482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.672130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.672412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.672685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.673384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.675318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.705539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1121/2255] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:05 2022] POST /identity/v3/projects => generated 373 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:05.706870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.718123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f66fff9-bceb-408f-a03e-7cd732eb8550 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.779932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.780607 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.780888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.781171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.781851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.783817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.809116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1135/2256] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:05 2022] POST /identity/v3/users => generated 287 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:05.810364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.822512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9cbe574-cd5f-45f7-a4d0-04a2a1b712ad admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.827256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.827917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.828204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.828485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.829208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.830849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.841589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1122/2257] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:05 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:05.845634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.856443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fc84b12e-a9e6-4a88-9906-be2e7ee0a15d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.861159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.861821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.862107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.862380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.863772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.865399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-000cc610-d0f1-49f8-aef3-734e44540fd0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.875887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1136/2258] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:05 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:05.880282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:05.890427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-026630c3-6342-4815-9fe5-068b26babf96 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:05.895330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:05.896214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.896499 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.896779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] PATH_INFO: `/v3/projects/a1002e5fea6c46a49cd5a57ec878e39f/users/6749ff3915854410b3f942e22face365/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.912314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=a1002e5fea6c46a49cd5a57ec878e39f, user_id=6749ff3915854410b3f942e22face365, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:05.915016 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:05.924037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:05.930437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1123/2259] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:05 2022] PUT /identity/v3/projects/a1002e5fea6c46a49cd5a57ec878e39f/users/6749ff3915854410b3f942e22face365/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:05.942671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:05.943073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:05.943689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:05.970558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] MFA Rules not processed for user `6749ff3915854410b3f942e22face365`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:05.971808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:06.016435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:06.039996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1137/2260] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:05 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:06.044588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.118951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0071fa0-92f4-4a78-afd1-f4a5ea9bedbf admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.123771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.124436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.124721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.125001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.125707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.127658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.155373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1124/2261] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:06 2022] POST /identity/v3/projects => generated 371 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:06.159544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.172780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c433844-c1d9-4667-ad79-2b151c23d2e6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.236793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.237470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.237758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.238584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.239672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.241598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.265334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1138/2262] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:06 2022] POST /identity/v3/users => generated 292 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:06.266758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.277509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b67e3f55-8fee-4435-8842-cbe679ba4522 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.282691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.283462 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.283765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.283979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.284702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.286475 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.299758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1125/2263] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:06 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:06.311730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.322902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89080d53-ebf2-4d51-94da-53e3b26c4fdb admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.329806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.331290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.331582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.331862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] PATH_INFO: `/v3/projects/c11287dd395641c5b94728524c80beb0/users/8b8c728c665f4e40b9925103b9381c6a/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.355528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c11287dd395641c5b94728524c80beb0, user_id=8b8c728c665f4e40b9925103b9381c6a, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.358503 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.377171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1139/2264] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:06 2022] PUT /identity/v3/projects/c11287dd395641c5b94728524c80beb0/users/8b8c728c665f4e40b9925103b9381c6a/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:06.387982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.407590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a6a7d97-52ac-4c3d-98e0-d20ab4e18079 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.490697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.491435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.491726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.492011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.492755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.494380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c66efe3-dc2a-4d93-85e5-5e36a71721e4 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.513773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1126/2265] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:06 2022] GET /identity/v3/roles => generated 1420 bytes in 126 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:06.517931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.530390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8f2d53a-9ae1-4445-b014-547aa5c16038 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.537643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.538888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.539355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.539944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] PATH_INFO: `/v3/projects/c11287dd395641c5b94728524c80beb0/users/8b8c728c665f4e40b9925103b9381c6a/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.542598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c11287dd395641c5b94728524c80beb0, user_id=8b8c728c665f4e40b9925103b9381c6a, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.546324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.559509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1140/2266] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:06 2022] PUT /identity/v3/projects/c11287dd395641c5b94728524c80beb0/users/8b8c728c665f4e40b9925103b9381c6a/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:06.570413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.570870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.571228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.601149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] MFA Rules not processed for user `8b8c728c665f4e40b9925103b9381c6a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:06.602529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:06.670838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:06.683083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1127/2267] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:06 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:06.692766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:06.694934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:06.754191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccc9d9ae-f689-4728-b439-32070e7210f7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:06.757926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:06.758574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:06.758889 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:06.759168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:06.768368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:06.769817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-686d22bd-541c-479e-a001-06aa8f959af8 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:06.779527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1141/2268] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:06 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:07.190889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:07.258757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4eb0aa53-5aa0-4d6b-9023-066c12004adf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:07.263585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:07.264590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:07.264877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:07.265172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] PATH_INFO: `/v3/users/6749ff3915854410b3f942e22face365` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:07.266386 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] RBAC: Authorizing `identity:delete_user(user_id=6749ff3915854410b3f942e22face365)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:07.268418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:07.323372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6749ff3915854410b3f942e22face365'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:07.332480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6749ff3915854410b3f942e22face365'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:07.333357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6749ff3915854410b3f942e22face365'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:07.342077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6749ff3915854410b3f942e22face365'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:07.360711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6749ff3915854410b3f942e22face365'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:07.368915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1128/2269] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:07 2022] DELETE /identity/v3/users/6749ff3915854410b3f942e22face365 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:07.373600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:07.373886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:07.374200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:07.397642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:07.404261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:07.449858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:07.483158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1142/2270] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:07.494259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:07.495761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:07.574159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df818867-2ee8-48da-8512-25d41111f30e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:07.579542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:07.580240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:07.580449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:07.580690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:07.594260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:07.595986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:07.609003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1129/2271] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:07 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 115 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:07.905498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:07.907063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:07.997581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41f3f7d-6684-4570-81e8-d276585ff870 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:08.005952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:08.006611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:08.006925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:08.007200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:08.009944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:08.068808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:08.070279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:08.084587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f29e287-a0bc-45db-9121-a6b1c3f55b6e service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:08.090041 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:08.091702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:08.091994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:08.092277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] PATH_INFO: `/v3/projects/a1002e5fea6c46a49cd5a57ec878e39f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:08.093555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:08.093848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] RBAC: Authorizing `identity:delete_project(project_id=a1002e5fea6c46a49cd5a57ec878e39f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:08.095229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:08.095800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:08.108731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1143/2272] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:07 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 204 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:08.111986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:08.162688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a1002e5fea6c46a49cd5a57ec878e39f', 'request_id': 'req-0d57ad30-e1b9-4698-a66e-cab4750a320a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.171642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a1002e5fea6c46a49cd5a57ec878e39f', 'request_id': 'req-0d57ad30-e1b9-4698-a66e-cab4750a320a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.180836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a1002e5fea6c46a49cd5a57ec878e39f', 'request_id': 'req-0d57ad30-e1b9-4698-a66e-cab4750a320a'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.205500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] The token cache is being invalidate because project a1002e5fea6c46a49cd5a57ec878e39f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:08.206213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.206461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.210189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1130/2273] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:08 2022] DELETE /identity/v3/projects/a1002e5fea6c46a49cd5a57ec878e39f => generated 0 bytes in 142 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:08.214447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:08.225608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-921a8afe-8ffb-41a9-89bd-8f34ed10c396 service nova] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:08.294936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:08.296176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:08.296736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:08.297245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] PATH_INFO: `/v3/users/8b8c728c665f4e40b9925103b9381c6a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:08.298726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8b8c728c665f4e40b9925103b9381c6a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:08.301653 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:08.327548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=98,keystone:DELETE=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:08.360430 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8b8c728c665f4e40b9925103b9381c6a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.370577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8b8c728c665f4e40b9925103b9381c6a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.371986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8b8c728c665f4e40b9925103b9381c6a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.383894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8b8c728c665f4e40b9925103b9381c6a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.384827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-98c24159-b7fb-4e21-bcda-7c07f2d0dca3 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8b8c728c665f4e40b9925103b9381c6a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.413807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1144/2274] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:08 2022] DELETE /identity/v3/users/8b8c728c665f4e40b9925103b9381c6a => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:08.748282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:08.749709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:08.763294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d57ad30-e1b9-4698-a66e-cab4750a320a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:08.832778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:08.833837 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:08.834135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:08.834414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] PATH_INFO: `/v3/projects/c11287dd395641c5b94728524c80beb0` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:08.835992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c11287dd395641c5b94728524c80beb0)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:08.837984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:08.887158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c11287dd395641c5b94728524c80beb0', 'request_id': 'req-9830e7a3-fc3f-4217-8084-d12b6f0293b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.892678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c11287dd395641c5b94728524c80beb0', 'request_id': 'req-9830e7a3-fc3f-4217-8084-d12b6f0293b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.900281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c11287dd395641c5b94728524c80beb0', 'request_id': 'req-9830e7a3-fc3f-4217-8084-d12b6f0293b8'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.912458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] The token cache is being invalidate because project c11287dd395641c5b94728524c80beb0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:08.912741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.913412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9830e7a3-fc3f-4217-8084-d12b6f0293b8 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:08.915284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1131/2275] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:08 2022] DELETE /identity/v3/projects/c11287dd395641c5b94728524c80beb0 => generated 0 bytes in 167 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:08.919957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:08.920505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:08.920630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:08.943152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:08.944717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:09.010727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1145/2276] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:09.021513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.021907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.022259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.042290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:09.043605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:09.051738 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1132/2277] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:09.055980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.065663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d1c8fee-d785-41f5-a54d-dcd56da09be4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.071891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.072560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.072845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.073126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.073883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:09.074177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.075827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.088262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1146/2278] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:06:09.092666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.102381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece3f9b4-763d-47ce-9656-dfaa7f903a29 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.107113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.107803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.108084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.108384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.109064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.111053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.142057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1133/2279] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/projects => generated 365 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:09.143624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.155718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-880ea735-81f0-4e03-8af3-8da3c4ed9757 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.220566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.221250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.221537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.222101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.223124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.225063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.251962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1147/2280] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/users => generated 283 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:09.253404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.259115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:09.265858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-549edb9d-d2f3-4562-9d6a-88eab9bd1b74 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.272368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.273268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.274001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.274001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.274700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.276354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.288129 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1134/2281] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:09.292519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.303296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b176bdcf-a843-4ccf-9244-282f7b364971 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.308312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.308976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.309259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.309537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.310262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.311904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cff4d27c-868a-438d-b03a-26d178961ad9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.323494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1148/2282] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:09.327720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.338445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206b0263-7d1b-47ff-a7ba-c6f4d8c5a143 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.343090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.343975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.344259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.344541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] PATH_INFO: `/v3/projects/3060016e22954de59f1ee77a6c92f0f5/users/6e451ab8c1c841d7b0735e7158646d85/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.359432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3060016e22954de59f1ee77a6c92f0f5, user_id=6e451ab8c1c841d7b0735e7158646d85, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.362114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.376084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1135/2283] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:09 2022] PUT /identity/v3/projects/3060016e22954de59f1ee77a6c92f0f5/users/6e451ab8c1c841d7b0735e7158646d85/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:09.387601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa615337-dd47-4333-9bb0-8a34c788b6c2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.387601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa615337-dd47-4333-9bb0-8a34c788b6c2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.387833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa615337-dd47-4333-9bb0-8a34c788b6c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.393732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=8,keystone:SELECT=88,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:09.424831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-fa615337-dd47-4333-9bb0-8a34c788b6c2 None None] MFA Rules not processed for user `6e451ab8c1c841d7b0735e7158646d85`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:09.426580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fa615337-dd47-4333-9bb0-8a34c788b6c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:09.497433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1149/2284] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3232 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:09.502008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.558817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.559100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.559378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.577531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] MFA Rules not processed for user `8f471531dac249d994f864102f7b9d3f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:09.578969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:09.596218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1809732-408f-4aff-9923-54bed23a2fb5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.602017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.602702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.603012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.603293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.603997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.605912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.631794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1136/2285] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/projects => generated 365 bytes in 130 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:09.635766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.636188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1150/2286] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3252 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:09.700011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1f66b6c-9258-4cf9-9565-c60dc816da1b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.704733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.705400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.705690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.705971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.706741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.709218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.729917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1137/2287] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:09.733821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.745318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8aae9856-4765-4b10-b185-9d26c5412f05 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.751560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.752353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.752759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.753155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.755531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.757198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.767976 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1151/2288] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:09.773577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.784225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd93a9a-73ac-41da-aec2-f6c15a1f13eb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.789510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.790461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.790943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.791225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] PATH_INFO: `/v3/projects/39c00463e7dc42bf9d3f4895edc24f94/users/788707290aad48c899ddaf23686a4c63/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.806548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] RBAC: Authorizing `identity:create_grant(project_id=39c00463e7dc42bf9d3f4895edc24f94, user_id=788707290aad48c899ddaf23686a4c63, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.809913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.824001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1138/2289] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:09 2022] PUT /identity/v3/projects/39c00463e7dc42bf9d3f4895edc24f94/users/788707290aad48c899ddaf23686a4c63/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:09.828043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.839100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9f6bff08-0aea-4b06-974d-a51acca38768 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.897761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.898432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.898708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.899014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.899735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.901351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.911938 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1152/2290] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:09.916046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.926897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-511a1a24-06a7-4db1-ad22-840824bc9afe admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:09.931607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:09.932496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.932787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.933066 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] PATH_INFO: `/v3/projects/39c00463e7dc42bf9d3f4895edc24f94/users/788707290aad48c899ddaf23686a4c63/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.935102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] RBAC: Authorizing `identity:create_grant(project_id=39c00463e7dc42bf9d3f4895edc24f94, user_id=788707290aad48c899ddaf23686a4c63, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:09.937751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9e7dea71-bf88-40dd-b0ef-cc00f633c2da admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:09.950056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1139/2291] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:09 2022] PUT /identity/v3/projects/39c00463e7dc42bf9d3f4895edc24f94/users/788707290aad48c899ddaf23686a4c63/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:09.955429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:09.957679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:09.960234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:09.960532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:09.960816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:09.967502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:09.985953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:09.989745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] MFA Rules not processed for user `788707290aad48c899ddaf23686a4c63`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:09.990987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.018155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9577216c-bde3-4ec2-a4f5-c96bfb9ad2ac admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:10.059237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1140/2292] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:09 2022] POST /identity/v3/auth/tokens => generated 3299 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:10.070665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:10.071007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:10.072219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.072359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:10.072518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:10.072799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:10.074728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.086652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:10.088219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:10.100320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1153/2293] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:09 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 145 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:10.132913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b06f3961-aa15-4524-8c1b-dacc56376e12 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:10.136615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:10.137257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:10.137535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:10.137839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:10.147454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:10.148885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:10.158171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1141/2294] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:10 2022] GET /identity/v3/auth/tokens => generated 3299 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:10.463480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:10.468019 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.567742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-be788094-a23a-4bc1-b0f4-77e9272a2abe service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:10.573620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:10.574237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:10.574523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:10.574858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:10.576715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.590394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:10.592061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98c06786-d3eb-417e-b33b-aeaca246e2e5 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:10.603064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1154/2295] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:10 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 140 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:10.654263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:10.752607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5db485d7-a3a2-4a7d-949b-5e11ab50d1df service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:10.757618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:10.758619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:10.759152 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:10.759496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] PATH_INFO: `/v3/users/6e451ab8c1c841d7b0735e7158646d85` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:10.760970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] RBAC: Authorizing `identity:delete_user(user_id=6e451ab8c1c841d7b0735e7158646d85)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:10.763107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:10.823042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:10.835621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6e451ab8c1c841d7b0735e7158646d85'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:10.844624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6e451ab8c1c841d7b0735e7158646d85'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:10.845702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6e451ab8c1c841d7b0735e7158646d85'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:10.854970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6e451ab8c1c841d7b0735e7158646d85'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:10.879028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6e451ab8c1c841d7b0735e7158646d85'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:10.887553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1142/2296] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:10 2022] DELETE /identity/v3/users/6e451ab8c1c841d7b0735e7158646d85 => generated 0 bytes in 234 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:10.891972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:10.892260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:10.892602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:10.912644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:10.914034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:10.994888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1155/2297] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:10 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:11.007725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:11.009526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:11.090350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2de7c428-b701-4036-b410-c39740e2eb2c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:11.094693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:11.095617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:11.095976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:11.096932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:11.108445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:11.109888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:11.120622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1143/2298] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:11.501717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:11.503177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:11.515807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-92d8abd0-a181-4503-ba83-4abe416beba7 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:11.521634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:11.522509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:11.522827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:11.523094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] PATH_INFO: `/v3/projects/3060016e22954de59f1ee77a6c92f0f5` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:11.524311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3060016e22954de59f1ee77a6c92f0f5)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:11.526252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:11.533068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:11.576989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3060016e22954de59f1ee77a6c92f0f5', 'request_id': 'req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.585847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3060016e22954de59f1ee77a6c92f0f5', 'request_id': 'req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.591776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3060016e22954de59f1ee77a6c92f0f5', 'request_id': 'req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.603541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] The token cache is being invalidate because project 3060016e22954de59f1ee77a6c92f0f5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:11.603828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.607099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.608844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1156/2299] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:11 2022] DELETE /identity/v3/projects/3060016e22954de59f1ee77a6c92f0f5 => generated 0 bytes in 108 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:11.613049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:11.623308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed06778-5ec1-410a-9c81-e24c9ff398aa service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:11.679279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:11.680149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:11.680430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:11.680709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] PATH_INFO: `/v3/users/788707290aad48c899ddaf23686a4c63` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:11.682030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] RBAC: Authorizing `identity:delete_user(user_id=788707290aad48c899ddaf23686a4c63)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:11.683986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:11.729215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '788707290aad48c899ddaf23686a4c63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.736668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '788707290aad48c899ddaf23686a4c63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.737333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '788707290aad48c899ddaf23686a4c63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.739538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=85,keystone:DELETE=13,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:11.745968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '788707290aad48c899ddaf23686a4c63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.758538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f7e30826-ae94-4804-9101-1a7332dae86d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '788707290aad48c899ddaf23686a4c63'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:11.764817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1144/2300] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:11 2022] DELETE /identity/v3/users/788707290aad48c899ddaf23686a4c63 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:12.051628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.054756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:12.069581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d2f2e6b-e2a9-4437-adaa-9269c4e27c32 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.137564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.138709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.139023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.139198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] PATH_INFO: `/v3/projects/39c00463e7dc42bf9d3f4895edc24f94` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.140383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] RBAC: Authorizing `identity:delete_project(project_id=39c00463e7dc42bf9d3f4895edc24f94)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.142369 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.196098 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '39c00463e7dc42bf9d3f4895edc24f94', 'request_id': 'req-99504a7d-e0ca-401c-a952-5495e89d7a03'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:12.204335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '39c00463e7dc42bf9d3f4895edc24f94', 'request_id': 'req-99504a7d-e0ca-401c-a952-5495e89d7a03'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:12.210508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '39c00463e7dc42bf9d3f4895edc24f94', 'request_id': 'req-99504a7d-e0ca-401c-a952-5495e89d7a03'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:12.222100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] The token cache is being invalidate because project 39c00463e7dc42bf9d3f4895edc24f94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:12.222410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:12.223105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-99504a7d-e0ca-401c-a952-5495e89d7a03 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:12.225191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1157/2301] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:12 2022] DELETE /identity/v3/projects/39c00463e7dc42bf9d3f4895edc24f94 => generated 0 bytes in 174 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:12.230614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.231038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.231512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.250015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:12.251857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:12.326684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1145/2302] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:12.328906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.329399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.329678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.349289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:12.350677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:12.360353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1158/2303] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:12.365318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.375256 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5df55c2f-c847-4de2-b1cc-78608472475e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.382695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.383398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.383686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.383966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.384697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:12.385010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.386615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.399274 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1146/2304] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:12 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:06:12.403681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.413898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d73e0bc7-f630-4495-b261-2fd83f3d85bc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.419243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.419668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.419958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.420233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.420922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.422897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.437829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=83,keystone:DELETE=14,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:12.453260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1159/2305] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/projects => generated 369 bytes in 50 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:12.457449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.468693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26d23686-df17-4a1e-8b7c-2d149cb7a447 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.537533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.538327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.538678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.539056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.539813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.541847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.565343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1147/2306] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/users => generated 285 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:12.566551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.578206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3cd0f2db-99a8-4e8a-9f88-84305ad4ec6a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.582958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.583723 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.584248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.584591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.585390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.587087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.597525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1160/2307] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:12.602195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.612364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba2951aa-8c0c-4236-b3da-b5368ab5b6d1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.617845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.618519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.618839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.619126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.619861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.621461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57b7d428-3c05-49d2-8d29-e9c9bd4eb217 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.632265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1148/2308] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:12 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:12.636606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.648496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-22f1346a-5929-452c-bf99-212fbd547ef7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.653844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.654965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.655316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.655828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] PATH_INFO: `/v3/projects/c6611c311aba4ad0b60bead01b238565/users/f5f3ce1f5c284cdba07b43f5238525cc/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.680648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c6611c311aba4ad0b60bead01b238565, user_id=f5f3ce1f5c284cdba07b43f5238525cc, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.683527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.698860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1161/2309] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:12 2022] PUT /identity/v3/projects/c6611c311aba4ad0b60bead01b238565/users/f5f3ce1f5c284cdba07b43f5238525cc/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:12.709230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.710145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.710815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.738533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] MFA Rules not processed for user `f5f3ce1f5c284cdba07b43f5238525cc`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:12.740623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:12.774049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=2,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:12.820572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1149/2310] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:12.824811 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.902343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c983875-6b5f-4f4e-bc5d-91861ddde2f6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:12.907205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:12.907882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:12.908161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:12.908434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:12.909132 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:12.911090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:12.914937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.916305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:12.944618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1162/2311] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/projects => generated 369 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:12.948759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:12.980957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28cd92ed-8feb-438c-9573-b72aefea84a0 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.017301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-089e24e8-9c9c-4aa0-9e0f-da67511f47cc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.023198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.024383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.024383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.024632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.025351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.027694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.034948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.035548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.035829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.036110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.038085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.049030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.050675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1163/2312] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:12 2022] POST /identity/v3/users => generated 291 bytes in 102 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:13.050907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.056145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.063864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1150/2313] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:12 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 149 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:13.067632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e7218189-11c9-4a25-bd3e-d50d728d89b6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.073355 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.074611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.074929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.075237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.075975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.077621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.089036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1164/2314] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:13 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:13.093064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.105213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d07039da-9308-49a5-8782-f0a827701e6f service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.110666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.111924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.112212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.112488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] PATH_INFO: `/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8/users/6435ddd244594aeab61f3577dc12395f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.128196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ebbd85c6bbd42d6b442c88af0ff59a8, user_id=6435ddd244594aeab61f3577dc12395f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.131037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.144357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1151/2315] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:13 2022] PUT /identity/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8/users/6435ddd244594aeab61f3577dc12395f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:13.149153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.159230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0ccbd78-b011-467a-a96e-d0e7491ad971 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.183368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:13.222828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.223500 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.223839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.224185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.224903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.226491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-30eb8a95-2c10-4996-a575-d59905c45a27 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.239867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1165/2316] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:13 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:13.240885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.252441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5ad4185f-f92d-41ba-85a4-8eec1bfb13e9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.258090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.259095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.259380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.259656 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] PATH_INFO: `/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8/users/6435ddd244594aeab61f3577dc12395f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.261613 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8ebbd85c6bbd42d6b442c88af0ff59a8, user_id=6435ddd244594aeab61f3577dc12395f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.264349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.276044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1152/2317] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:13 2022] PUT /identity/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8/users/6435ddd244594aeab61f3577dc12395f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:13.288103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.288388 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.288663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.315743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] MFA Rules not processed for user `6435ddd244594aeab61f3577dc12395f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:13.316949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.387204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1166/2318] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:13 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:13.402437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.404198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.466376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-808a7df3-b9a6-46b1-a546-6e927158a5b1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.470730 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.471411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.471705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.471984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.480275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:13.484096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.485948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.506943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1153/2319] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:13 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:13.548152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.549582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.571498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29365164-08dd-4c71-bd0e-e664218b2d82 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.587069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.587742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.588028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.588302 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.676333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.679087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.694310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1167/2320] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:13 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 146 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:13.803112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:13.804960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.919996 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ce709f5-b0c8-446a-aef3-edc6b68218ce service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:13.933137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:13.934250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:13.934544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:13.934851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:13.936733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:13.955108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:13.957440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6ed3eac8-38e8-43fe-a895-fbfe623c3f52 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:13.982571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1154/2321] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:13 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 181 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:17.298580 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:17.363123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:17.388118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cc15182-9193-4624-8024-1dc31bbcd5fe service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:17.400496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:17.401416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:17.401703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:17.402133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] PATH_INFO: `/v3/users/f5f3ce1f5c284cdba07b43f5238525cc` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:17.403461 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] RBAC: Authorizing `identity:delete_user(user_id=f5f3ce1f5c284cdba07b43f5238525cc)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:17.405540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:17.478171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'f5f3ce1f5c284cdba07b43f5238525cc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:17.490189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'f5f3ce1f5c284cdba07b43f5238525cc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:17.491031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'f5f3ce1f5c284cdba07b43f5238525cc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:17.499399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'f5f3ce1f5c284cdba07b43f5238525cc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:17.500841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'f5f3ce1f5c284cdba07b43f5238525cc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:17.527799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1168/2322] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:17 2022] DELETE /identity/v3/users/f5f3ce1f5c284cdba07b43f5238525cc => generated 0 bytes in 227 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:17.529903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:17.530740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:17.531058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:17.578791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:17.580038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:17.663121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1155/2323] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:17.672528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:17.674215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:17.752586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d89f82f3-325b-47b3-9cfb-64ec4a81dc5f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:17.757042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:17.757728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:17.758009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:17.758283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:17.769761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:17.771452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:17.791958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1169/2324] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:17.993398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:17.994177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:18.084804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dea45133-62af-44e9-8134-3afa883a3f04 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:18.092688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:18.093550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:18.096286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:18.096286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:18.097348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:18.115254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:18.121834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:18.123371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:18.153289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-305bcb7d-eeb5-443f-9451-268737bbc563 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:18.167847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:18.168966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:18.169086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:18.169275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] PATH_INFO: `/v3/projects/c6611c311aba4ad0b60bead01b238565` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:18.171105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c6611c311aba4ad0b60bead01b238565)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:18.175517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:18.195998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:18.198102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:18.219011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1156/2325] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:17 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 227 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:18.263787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c6611c311aba4ad0b60bead01b238565', 'request_id': 'req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.273620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c6611c311aba4ad0b60bead01b238565', 'request_id': 'req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.279966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=82,keystone:DELETE=16,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:18.296444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c6611c311aba4ad0b60bead01b238565', 'request_id': 'req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.316639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] The token cache is being invalidate because project c6611c311aba4ad0b60bead01b238565 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:18.318664 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.319786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.326949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1170/2326] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:18 2022] DELETE /identity/v3/projects/c6611c311aba4ad0b60bead01b238565 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:18.327298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:18.341155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-293d1785-3861-4129-863c-f6cbf1be64f4 service nova] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:18.418151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:18.418675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:18.418964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:18.419276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] PATH_INFO: `/v3/users/6435ddd244594aeab61f3577dc12395f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:18.421278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] RBAC: Authorizing `identity:delete_user(user_id=6435ddd244594aeab61f3577dc12395f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:18.423544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:18.482907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '6435ddd244594aeab61f3577dc12395f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.490757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '6435ddd244594aeab61f3577dc12395f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.491409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '6435ddd244594aeab61f3577dc12395f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.498791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '6435ddd244594aeab61f3577dc12395f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.517592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-aa03c744-6c67-4715-910e-39a0688b7c1f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '6435ddd244594aeab61f3577dc12395f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:18.526512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1157/2327] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:18 2022] DELETE /identity/v3/users/6435ddd244594aeab61f3577dc12395f => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:18.882970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:18.884458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:18.901752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cace26f0-df1d-4385-8b9b-0cef2d64f2c1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:18.987571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:18.989202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:18.989507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:18.989790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] PATH_INFO: `/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:18.991663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8ebbd85c6bbd42d6b442c88af0ff59a8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:18.993624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.073368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8ebbd85c6bbd42d6b442c88af0ff59a8', 'request_id': 'req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:19.084765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8ebbd85c6bbd42d6b442c88af0ff59a8', 'request_id': 'req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:19.097359 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8ebbd85c6bbd42d6b442c88af0ff59a8', 'request_id': 'req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:19.117045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] The token cache is being invalidate because project 8ebbd85c6bbd42d6b442c88af0ff59a8 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:19.117375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:19.118046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-cb08b9aa-05ca-4605-baf8-2940d8c7d6a0 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:19.120186 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1171/2328] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:18 2022] DELETE /identity/v3/projects/8ebbd85c6bbd42d6b442c88af0ff59a8 => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:19.125057 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.125586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.125681 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.144615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:19.146713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.188277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:19.235224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1158/2329] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:19.236923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.237249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.237467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.262826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:19.264077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.273080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1172/2330] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 37 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:19.277243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.287077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-80f6fc4e-f88b-405d-b6e0-d2495307a6d1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.292244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.292993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.293289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.293680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.298813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:19.299162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.300819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.318011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1159/2331] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:06:19.322178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.331592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5c97c85-de39-42a3-ae6c-3c951cd43f74 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.338176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.338628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.338797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.339070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.339765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.341836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.369274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1173/2332] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:19 2022] POST /identity/v3/projects => generated 371 bytes in 48 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:19.373489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.384490 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4988932f-3780-4b5a-b85f-383963dcd2b8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.449647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.450333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.450621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.450933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.451640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.453585 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.477561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1160/2333] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:19 2022] POST /identity/v3/users => generated 286 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:19.481397 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.492954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-89b75bd8-c144-4337-b3ce-ba49533d7a2b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.498815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.499630 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.499907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.500310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.501360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.503133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.521908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1174/2334] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:19 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:19.522592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.535234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc50ad27-b593-418e-86e0-98b50b5ada8b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.541140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.541821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.542110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.542390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.543157 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.544774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-88688d27-13dd-4af5-a806-78625d5050a0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.559375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1161/2335] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:19 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:19.560731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.581348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301285ca-275f-4233-b9a0-cb086456a596 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.587965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.589242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.590244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.590647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] PATH_INFO: `/v3/projects/aa1d77fb74474a5f9e0291d9599e2b35/users/0da74a30a8aa46919ccafc871326f190/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.613843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] RBAC: Authorizing `identity:create_grant(project_id=aa1d77fb74474a5f9e0291d9599e2b35, user_id=0da74a30a8aa46919ccafc871326f190, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.617403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.658674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1175/2336] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:19 2022] PUT /identity/v3/projects/aa1d77fb74474a5f9e0291d9599e2b35/users/0da74a30a8aa46919ccafc871326f190/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:19.663748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.664115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.664469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.709485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:19.710808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.741031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.742698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.774431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=8,keystone:SELECT=89,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:19.874883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1162/2337] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 210 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:19.885237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:19.888273 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.904381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1b6ed1-782e-4633-acec-288121d68b55 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:19.913715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:19.914741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:19.915303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:19.915794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:19.918833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:19.949069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:19.951150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c5a5988-b25b-4e62-bacf-1cfa21203436 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:19.964497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1176/2338] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:19 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 224 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:20.003091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:20.027140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76a954e2-df0e-45e6-b972-fb407e63c9aa None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:20.049828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:20.051411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:20.052949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:20.053342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:20.069960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:20.072935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:20.086890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1163/2339] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:20.992267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:20.992765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:20.992878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.023011 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] MFA Rules not processed for user `0da74a30a8aa46919ccafc871326f190`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:21.024320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:21.101424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1177/2340] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:20 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:21.105980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.116242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09cd82e7-613c-45a9-b046-9a47e0c44fb7 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.120944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.121699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.122064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.122419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.123222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.125225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.151743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1164/2341] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:21 2022] POST /identity/v3/projects => generated 371 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:21.155786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.167002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4bc36d20-3e9e-46c6-bb8d-4ab1c4a012a2 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.224005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.224675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.224963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.225245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.225927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.227905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.248746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1178/2342] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:21 2022] POST /identity/v3/users => generated 292 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:21.252827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.264234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac111b7-a45f-4b22-b9ca-3eeb87f24424 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.268890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.269548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.269861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.270139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.272426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.272667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.284604 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1165/2343] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:21 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:21.289195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.299852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c490a24-9e0b-421a-98e8-f5a5cd349515 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.305649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.307054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.307339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.307614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] PATH_INFO: `/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65/users/9b7ead61525649a38f564a4aa916fe9b/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.324199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] RBAC: Authorizing `identity:create_grant(project_id=96da00f9d0aa4d93a0154c69fdf91a65, user_id=9b7ead61525649a38f564a4aa916fe9b, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.326914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.329980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:21.340939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1179/2344] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:21 2022] PUT /identity/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65/users/9b7ead61525649a38f564a4aa916fe9b/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:21.344987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.356098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3408b0e-887b-4242-9f0e-8264b3bc4f70 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.412667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.413325 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.413615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.413901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.414624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.416268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-287f02e7-428c-4793-a626-db41026fe0c9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.430320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1166/2345] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:21 2022] GET /identity/v3/roles => generated 1420 bytes in 86 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:21.431588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:21.442168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-75feb1cd-1a57-4df2-8949-16355df84681 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:21.447109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:21.447997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:21.448283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:21.448562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] PATH_INFO: `/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65/users/9b7ead61525649a38f564a4aa916fe9b/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:21.450528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] RBAC: Authorizing `identity:create_grant(project_id=96da00f9d0aa4d93a0154c69fdf91a65, user_id=9b7ead61525649a38f564a4aa916fe9b, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:21.453268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:21.463402 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1180/2346] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:21 2022] PUT /identity/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65/users/9b7ead61525649a38f564a4aa916fe9b/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:22.148878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:22.149440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:22.149743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:22.177322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] MFA Rules not processed for user `9b7ead61525649a38f564a4aa916fe9b`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:22.178751 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:22.257415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1167/2347] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:22 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:22.269035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:22.270985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:22.329924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d375bc24-3f84-4947-a259-00cbef8f993b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:22.333874 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:22.334524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:22.334839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:22.335120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:22.391193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:22.397549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:22.402536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1181/2348] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:22 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:23.738423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:23.748749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5705f56e-a24b-412f-96f0-ecfac9161d1d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:23.758841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:23.759519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:23.759826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:23.760093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:23.769766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:23.771626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a921bf8e-fe63-4a6a-8570-f4409d383842 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:23.783563 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1168/2349] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:23 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 46 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:24.157469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:24.158947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:24.251338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a76484-201d-4dc5-895a-a9c89b6ef3a9 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:24.258224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:24.258838 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:24.259123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:24.259408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:24.261204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:24.275006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:24.276775 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-094b1122-1bdb-434f-a197-e20fbd919898 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:24.293202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1182/2350] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:24 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 136 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:26.700376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:26.700879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:26.700966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:26.722317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:26.723908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:26.738265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:26.765297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751e739-a324-43c7-a283-7d6010742981 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:26.765763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751e739-a324-43c7-a283-7d6010742981 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:26.765875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a751e739-a324-43c7-a283-7d6010742981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:26.783448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a751e739-a324-43c7-a283-7d6010742981 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:26.784713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1169/2351] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:06:26 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:26.784894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a751e739-a324-43c7-a283-7d6010742981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:26.792712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1183/2352] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:06:26 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:26.837782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:26.839283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:26.845004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a751e739-a324-43c7-a283-7d6010742981 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:26.846495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a751e739-a324-43c7-a283-7d6010742981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:26.867280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:26.918906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7464e751-a481-4c33-bdba-7b67bc741965 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:26.924050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:26.924700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:26.924988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:26.925267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:26.925376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a751e739-a324-43c7-a283-7d6010742981 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:26.930414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:26.931221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:26.931509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:26.931793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:26.934314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:26.935951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:26.940453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:26.942807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:26.945747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1170/2353] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:06:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:06:26.953345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1184/2354] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:06:26 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:06:36.713053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:36.716005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:36.810871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-179728ad-5dc9-4d9c-9244-57ed503839bd service swift] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:36.816850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:36.818526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:36.818890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:36.819116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] PATH_INFO: `/v3/users/ddda2bac031d4635b27421d22bb40fff` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:36.820347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ddda2bac031d4635b27421d22bb40fff)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:36.822807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:36.875770 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ddda2bac031d4635b27421d22bb40fff'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:36.896306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ddda2bac031d4635b27421d22bb40fff'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:36.897134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ddda2bac031d4635b27421d22bb40fff'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:36.899966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:36.909845 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ddda2bac031d4635b27421d22bb40fff'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:36.925074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ddda2bac031d4635b27421d22bb40fff'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:36.932454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1171/2355] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:36 2022] DELETE /identity/v3/users/ddda2bac031d4635b27421d22bb40fff => generated 0 bytes in 221 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:36.949009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=25 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:37.144852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:37.163611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e7031a8-b67d-44e8-bd1c-5d1fec45497a service swift] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:37.296498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:37.303783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:37.304351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:37.304845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] PATH_INFO: `/v3/projects/ee9d5b852be444f48a33b2f4be5c34be` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:37.321432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] RBAC: Authorizing `identity:delete_project(project_id=ee9d5b852be444f48a33b2f4be5c34be)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:37.323487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:37.455341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ee9d5b852be444f48a33b2f4be5c34be', 'request_id': 'req-0d91bc8d-32ab-4258-ae09-517a9650f87f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.471258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ee9d5b852be444f48a33b2f4be5c34be', 'request_id': 'req-0d91bc8d-32ab-4258-ae09-517a9650f87f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.480788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ee9d5b852be444f48a33b2f4be5c34be', 'request_id': 'req-0d91bc8d-32ab-4258-ae09-517a9650f87f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.496248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] The token cache is being invalidate because project ee9d5b852be444f48a33b2f4be5c34be was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:37.496868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.497780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.515553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1185/2356] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:37 2022] DELETE /identity/v3/projects/ee9d5b852be444f48a33b2f4be5c34be => generated 0 bytes in 368 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:37.516511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:37.518616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:37.548496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d6423f8-11ab-4aeb-87bb-21b0b4cc05d6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:37.642286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:37.644050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:37.644414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:37.644892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] PATH_INFO: `/v3/users/af119e9383b844eab16d30caa44ce9db` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:37.646638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] RBAC: Authorizing `identity:delete_user(user_id=af119e9383b844eab16d30caa44ce9db)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:37.648941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:37.733449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'af119e9383b844eab16d30caa44ce9db'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.742836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'af119e9383b844eab16d30caa44ce9db'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.743491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'af119e9383b844eab16d30caa44ce9db'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.758984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'af119e9383b844eab16d30caa44ce9db'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.773214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'af119e9383b844eab16d30caa44ce9db'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:37.780839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1172/2357] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:37 2022] DELETE /identity/v3/users/af119e9383b844eab16d30caa44ce9db => generated 0 bytes in 265 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:38.120128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:38.121963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:38.144085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d91bc8d-32ab-4258-ae09-517a9650f87f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:38.232119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:38.233028 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:38.233530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:38.234150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] PATH_INFO: `/v3/projects/c0a2ac49b37b459d9c979a47acbe488b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:38.235501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] RBAC: Authorizing `identity:delete_project(project_id=c0a2ac49b37b459d9c979a47acbe488b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:38.237689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:38.280181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:38.298481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c0a2ac49b37b459d9c979a47acbe488b', 'request_id': 'req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:38.313682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c0a2ac49b37b459d9c979a47acbe488b', 'request_id': 'req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:38.323551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c0a2ac49b37b459d9c979a47acbe488b', 'request_id': 'req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:38.343951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] The token cache is being invalidate because project c0a2ac49b37b459d9c979a47acbe488b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:38.344510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:38.345621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:38.365686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1186/2358] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:38 2022] DELETE /identity/v3/projects/c0a2ac49b37b459d9c979a47acbe488b => generated 0 bytes in 233 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:39.163669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:39.165153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:39.178329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4a4db2d3-f3de-4a01-ac12-b72a9cd9010b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:39.225552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:39.236308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:39.237193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:39.237477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:39.237783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] PATH_INFO: `/v3/users/0da74a30a8aa46919ccafc871326f190` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:39.240004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0da74a30a8aa46919ccafc871326f190)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:39.241924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:39.288631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0da74a30a8aa46919ccafc871326f190'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.295596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0da74a30a8aa46919ccafc871326f190'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.296206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0da74a30a8aa46919ccafc871326f190'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.303630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0da74a30a8aa46919ccafc871326f190'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.315372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0da74a30a8aa46919ccafc871326f190'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.322344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1173/2359] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:39 2022] DELETE /identity/v3/users/0da74a30a8aa46919ccafc871326f190 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:39.498922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:39.500463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:39.514185 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a25a2f65-8b19-4f5c-b88c-d028eb8aaa68 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:39.582280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:39.583099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:39.583384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:39.583663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] PATH_INFO: `/v3/projects/aa1d77fb74474a5f9e0291d9599e2b35` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:39.584848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] RBAC: Authorizing `identity:delete_project(project_id=aa1d77fb74474a5f9e0291d9599e2b35)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:39.586810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:39.633406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'aa1d77fb74474a5f9e0291d9599e2b35', 'request_id': 'req-2e937947-dae6-4450-aae7-8c4a48031546'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.640641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'aa1d77fb74474a5f9e0291d9599e2b35', 'request_id': 'req-2e937947-dae6-4450-aae7-8c4a48031546'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.646866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'aa1d77fb74474a5f9e0291d9599e2b35', 'request_id': 'req-2e937947-dae6-4450-aae7-8c4a48031546'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.657565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] The token cache is being invalidate because project aa1d77fb74474a5f9e0291d9599e2b35 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:39.657850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.658478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.660138 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1187/2360] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:39 2022] DELETE /identity/v3/projects/aa1d77fb74474a5f9e0291d9599e2b35 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:39.664250 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:39.665658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:39.678084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee106a3b-04c2-4cb6-8ded-9f5c554e5ec7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:39.732999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:39.733926 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:39.734240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:39.734516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] PATH_INFO: `/v3/users/9b7ead61525649a38f564a4aa916fe9b` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:39.735729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] RBAC: Authorizing `identity:delete_user(user_id=9b7ead61525649a38f564a4aa916fe9b)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:39.737636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:39.803270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9b7ead61525649a38f564a4aa916fe9b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.811993 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9b7ead61525649a38f564a4aa916fe9b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.813126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9b7ead61525649a38f564a4aa916fe9b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.821709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9b7ead61525649a38f564a4aa916fe9b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.830224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:39.836031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-16044025-2789-4b4d-ac36-328a211a3f7b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9b7ead61525649a38f564a4aa916fe9b'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:39.843324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1174/2361] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:39 2022] DELETE /identity/v3/users/9b7ead61525649a38f564a4aa916fe9b => generated 0 bytes in 180 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:40.099408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.101069 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:40.114345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e937947-dae6-4450-aae7-8c4a48031546 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.178917 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:40.192304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.193494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.193784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.194063 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] PATH_INFO: `/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.195310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] RBAC: Authorizing `identity:delete_project(project_id=96da00f9d0aa4d93a0154c69fdf91a65)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.197251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.245022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '96da00f9d0aa4d93a0154c69fdf91a65', 'request_id': 'req-71ef85fb-f54d-4a1d-97af-52eb812f0440'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:40.253781 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '96da00f9d0aa4d93a0154c69fdf91a65', 'request_id': 'req-71ef85fb-f54d-4a1d-97af-52eb812f0440'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:40.260224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '96da00f9d0aa4d93a0154c69fdf91a65', 'request_id': 'req-71ef85fb-f54d-4a1d-97af-52eb812f0440'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:40.271493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] The token cache is being invalidate because project 96da00f9d0aa4d93a0154c69fdf91a65 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:40.271779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:40.272433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-71ef85fb-f54d-4a1d-97af-52eb812f0440 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:40.273996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1188/2362] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:40 2022] DELETE /identity/v3/projects/96da00f9d0aa4d93a0154c69fdf91a65 => generated 0 bytes in 175 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:40.279212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.279688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.279828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.298219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:40.299449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:40.361779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1175/2363] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:40.370199 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.370486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.370782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.389713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:40.390463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:40.400870 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1189/2364] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:40.405903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.420648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f13784b8-50ae-4176-a825-034493a6ef34 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.425742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.427302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.427596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.427879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.429949 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:40.430265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.431915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.449183 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1176/2365] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:06:40.450530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.461822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-83effd0e-6b65-47e4-84c5-d20925d5d99f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.467288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.467972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.468258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.468531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.469239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.471289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.517007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1190/2366] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:40 2022] POST /identity/v3/projects => generated 383 bytes in 67 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:40.522123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.534173 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b5b4da1-446a-4a68-ae71-b54b8d71a850 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.593290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.593953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.594234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.594508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.595231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.597126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.617720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1177/2367] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:40 2022] POST /identity/v3/users => generated 292 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:40.621782 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.632299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f551ca6-a567-4a35-b9e4-2e3f192d1635 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.636946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.637603 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.637887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.638163 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.638919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.640518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.651378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1191/2368] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:40.655414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.666252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5908778-5319-4648-b46b-f026b4d02847 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.670883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.671542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.671843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.672109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.672828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.674531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b7e1023-ed7c-4e6b-957a-cc7962c04440 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.684466 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1178/2369] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:40 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:40.688534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.698835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a31ca98e-2341-4124-b294-122a10abb0ad admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.703731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.704616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.704905 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.705254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] PATH_INFO: `/v3/projects/b76aa922bde4477f95af271349afb03c/users/b6117963b7ae49d39f8094d53876d236/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.720273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=b76aa922bde4477f95af271349afb03c, user_id=b6117963b7ae49d39f8094d53876d236, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.723695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.736594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1192/2370] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:40 2022] PUT /identity/v3/projects/b76aa922bde4477f95af271349afb03c/users/b6117963b7ae49d39f8094d53876d236/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:40.748146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.748440 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.748793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.776303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] MFA Rules not processed for user `b6117963b7ae49d39f8094d53876d236`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:40.777514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:40.821866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=96,keystone:DELETE=1,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:40.843635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1179/2371] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:40 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:40.854735 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:40.856465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:40.912696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703da8f0-3060-41bf-9226-8f5ece47c0c1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:40.916309 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:40.916984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:40.917272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:40.917547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:40.925855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:40.927420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0523a6df-d51f-43b6-993b-f9b535221394 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:40.936268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1193/2372] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:40 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 82 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:48.595439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:48.667411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd203a91-34fe-40a0-9584-9633ce84dd6b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:48.672749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:48.673642 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:48.673933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:48.674214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] PATH_INFO: `/v3/users/b6117963b7ae49d39f8094d53876d236` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:48.675452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b6117963b7ae49d39f8094d53876d236)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:48.677444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:48.730107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b6117963b7ae49d39f8094d53876d236'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:48.740196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b6117963b7ae49d39f8094d53876d236'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:48.740842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b6117963b7ae49d39f8094d53876d236'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:48.748549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b6117963b7ae49d39f8094d53876d236'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:48.766938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b6117963b7ae49d39f8094d53876d236'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:48.774009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1180/2373] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:48 2022] DELETE /identity/v3/users/b6117963b7ae49d39f8094d53876d236 => generated 0 bytes in 179 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:48.778631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b473e2-b0eb-4e6f-91fa-2cbebd7ab8c6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:48.779487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b473e2-b0eb-4e6f-91fa-2cbebd7ab8c6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:48.779749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-12b473e2-b0eb-4e6f-91fa-2cbebd7ab8c6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:48.798586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-12b473e2-b0eb-4e6f-91fa-2cbebd7ab8c6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:48.799862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-12b473e2-b0eb-4e6f-91fa-2cbebd7ab8c6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:48.814051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:48.885657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1194/2374] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:48 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:48.898149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:48.899866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:48.968512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-449142f3-f75b-4345-9ff1-373939ee04a5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:48.972431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:48.973161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:48.974056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:48.974347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:48.985961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:48.987762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:48.999820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1181/2375] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:48 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 102 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:49.242254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:49.242542 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:49.242852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:49.265273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:49.266856 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:49.350730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1195/2376] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:06:49 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 110 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:49.364872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:49.366598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:49.380143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:49.476010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-753cb14e-6def-4d0e-96aa-418fd921b744 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:49.481914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:49.482633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:49.483335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:49.483335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:49.501415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:49.503420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cf90ab0b-1e95-4c51-86a3-4723ace15254 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:49.515965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1182/2377] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:06:49 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 152 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:06:49.775134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:49.776767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:49.799214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2f80497-ed6c-437f-93d3-4e9d22536154 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:49.815752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:49.817358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:49.817854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:49.818928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] PATH_INFO: `/v3/projects/b76aa922bde4477f95af271349afb03c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:49.822101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] RBAC: Authorizing `identity:delete_project(project_id=b76aa922bde4477f95af271349afb03c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:49.825809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:49.939310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'b76aa922bde4477f95af271349afb03c', 'request_id': 'req-752f7554-2639-4447-aaf2-9b8d316a35bb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:49.956046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'b76aa922bde4477f95af271349afb03c', 'request_id': 'req-752f7554-2639-4447-aaf2-9b8d316a35bb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:49.975788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'b76aa922bde4477f95af271349afb03c', 'request_id': 'req-752f7554-2639-4447-aaf2-9b8d316a35bb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:49.994846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] The token cache is being invalidate because project b76aa922bde4477f95af271349afb03c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:49.995711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:49.997395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-752f7554-2639-4447-aaf2-9b8d316a35bb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:50.006664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.007242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1196/2378] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:49 2022] DELETE /identity/v3/projects/b76aa922bde4477f95af271349afb03c => generated 0 bytes in 226 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:50.007686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.007970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.030417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:50.032342 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:50.101703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1183/2379] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:50.109806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.110123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.110441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.145304 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:50.147154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:50.156103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1197/2380] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 48 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:50.160618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.171804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aebc673-efbc-401d-a337-3996b0724bf7 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.177329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.178086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.178369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.178647 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.179406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:50.179718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.181343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.197638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1184/2381] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:50 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:06:50.201777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.212362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c5f3ed09-6ed8-46f2-a06f-aed9515ae4a3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.217221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.217943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.218228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.218507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.219250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.221223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.260006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1198/2382] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:50 2022] POST /identity/v3/projects => generated 383 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:06:50.261074 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.274581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38afed4f-0dfb-428d-972f-4344938d7bf2 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.357505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.358257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.358557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.358901 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.359815 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.361832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.392091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1185/2383] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:50 2022] POST /identity/v3/users => generated 292 bytes in 129 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:50.393434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.407690 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1aaaaf21-43f6-4f06-b5d7-9bb6557e57db admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.412785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.413602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.413895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.414174 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.415088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.416910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.429998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1199/2384] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:50.434191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.448614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f781b6e-5e04-4130-9688-d1d62599ec81 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.454877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.455735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.456118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.456471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.457328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.459134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-819e1877-e184-4f16-ba8e-82724dfc7009 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.470550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1186/2385] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:50 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:50.474847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.485274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23b90cf6-180c-46d8-b763-d75b9baff8b6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.491202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.492137 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.492417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.492692 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] PATH_INFO: `/v3/projects/25bdd17f5ef84af79ace4b4edefdcfcc/users/859d8ab3627c49b296110aa99db6fa01/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.496187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=91,keystone:DELETE=7,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:50.512518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] RBAC: Authorizing `identity:create_grant(project_id=25bdd17f5ef84af79ace4b4edefdcfcc, user_id=859d8ab3627c49b296110aa99db6fa01, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.515181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.531261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1200/2386] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:50 2022] PUT /identity/v3/projects/25bdd17f5ef84af79ace4b4edefdcfcc/users/859d8ab3627c49b296110aa99db6fa01/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:50.541495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.541794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.542082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.558997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:50.601402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] MFA Rules not processed for user `859d8ab3627c49b296110aa99db6fa01`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:50.602816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:50.719838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1187/2387] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:50 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 179 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:50.731802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:50.733556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:50.808798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9f6e4ba-aa39-4efb-9b03-ed018445cb69 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:50.813004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:50.813663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:50.813952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:50.814226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:50.824616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:50.826080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:50.836800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1201/2388] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:50 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 105 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:53.055577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:53.057025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:53.123694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a60209cb-9e0e-4726-b91e-731c0d07a6d7 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:53.128106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:53.128950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:53.129935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:53.130238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:53.132519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:53.227266 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:53.228825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c774c76-4f61-479e-ab58-25e6675aa33c service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:53.240078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1188/2389] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 185 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:53.465380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:53.478449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3636dc8b-af2a-4812-97c8-e257a19e7c74 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:53.484220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:53.485213 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:53.485565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:53.485909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] PATH_INFO: `/v3/users/859d8ab3627c49b296110aa99db6fa01` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:53.490269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] RBAC: Authorizing `identity:delete_user(user_id=859d8ab3627c49b296110aa99db6fa01)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:53.492836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:53.563848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '859d8ab3627c49b296110aa99db6fa01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:53.585732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '859d8ab3627c49b296110aa99db6fa01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:53.586560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '859d8ab3627c49b296110aa99db6fa01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:53.596863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '859d8ab3627c49b296110aa99db6fa01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:53.597694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '859d8ab3627c49b296110aa99db6fa01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:53.633794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1202/2390] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:53 2022] DELETE /identity/v3/users/859d8ab3627c49b296110aa99db6fa01 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:53.635661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:53.635982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:53.636283 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:53.682885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:53.684159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:53.742204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:53.774257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1189/2391] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:53 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:06:53.784055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:53.785512 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:53.855415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a13b28f6-4a76-4273-ab36-1d08e3a5a499 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:53.859270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:53.859886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:53.860327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:53.860612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:53.893226 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:53.894764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0eee188-5cce-43d7-aa99-8bf1c367de1e service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:53.906183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1203/2392] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:53 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 123 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:54.261239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:54.263425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:54.296008 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c03924b-1cba-43ec-9895-08b94d7f7d83 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:54.317122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:54.318950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.319450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.319952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] PATH_INFO: `/v3/projects/25bdd17f5ef84af79ace4b4edefdcfcc` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.323852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] RBAC: Authorizing `identity:delete_project(project_id=25bdd17f5ef84af79ace4b4edefdcfcc)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:54.331123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:54.441198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '25bdd17f5ef84af79ace4b4edefdcfcc', 'request_id': 'req-56d0615f-5670-46e7-b901-6a18b4c8222f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:54.456759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '25bdd17f5ef84af79ace4b4edefdcfcc', 'request_id': 'req-56d0615f-5670-46e7-b901-6a18b4c8222f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:54.469096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '25bdd17f5ef84af79ace4b4edefdcfcc', 'request_id': 'req-56d0615f-5670-46e7-b901-6a18b4c8222f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:54.483309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] The token cache is being invalidate because project 25bdd17f5ef84af79ace4b4edefdcfcc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:54.483632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:54.484363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-56d0615f-5670-46e7-b901-6a18b4c8222f admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:54.486005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1190/2393] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:54 2022] DELETE /identity/v3/projects/25bdd17f5ef84af79ace4b4edefdcfcc => generated 0 bytes in 225 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:54.491195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.491740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.491937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.510346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:INSERT=2,keystone:DELETE=10 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:54.535053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:54.536892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:54.649020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1204/2394] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 159 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:54.658280 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.658689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.659033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.689947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:54.691251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:54.700997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1191/2395] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:54 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 44 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:54.705081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:54.729792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3da50110-0492-4d99-841b-4f93b1dd9bb1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:54.734748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:54.735454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.735740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.739629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.740551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:54.740873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:54.744871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:54.760927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1205/2396] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:54 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 56 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:06:54.765125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:54.777164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffddf63b-0654-42e6-8efe-9acab335e5cf None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:54.789561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:54.790316 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.790605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.792662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.793424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:54.801595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:54.867978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1192/2397] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:54 2022] POST /identity/v3/projects => generated 369 bytes in 103 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:54.872037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:54.888407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-39e07647-943a-472a-8a7f-bc0689a6a26b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:54.992233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:54.992961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:54.993246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:54.993530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:54.994307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:54.996406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:55.020106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1206/2398] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:54 2022] POST /identity/v3/users => generated 285 bytes in 149 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:55.024799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:55.040828 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ac65f0b1-5f1c-4c0f-851c-33df3fab67d9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:55.046468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:55.047258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:55.047663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:55.048019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:55.049270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:55.051840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:55.077727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1193/2399] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:55.079215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:55.090405 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b31866da-e124-4318-b60c-b33752bb4521 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:55.095936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:55.097845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:55.098390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:55.099356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:55.100280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:55.103729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61ca239f-a40d-4af6-8ff7-ddb09be3a1a4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:55.118290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1207/2400] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:55 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:55.122716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:55.135383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-34542350-968a-4695-a78c-26b8fe183edc admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:55.141545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:55.143024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:55.143425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:55.144053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] PATH_INFO: `/v3/projects/ce0bcbe23a1d4854ad1565de299887c0/users/30aa83d0bc8c4b6ca1ea53ed6e9b8d4e/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:55.163297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] RBAC: Authorizing `identity:create_grant(project_id=ce0bcbe23a1d4854ad1565de299887c0, user_id=30aa83d0bc8c4b6ca1ea53ed6e9b8d4e, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:55.166617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:55.186108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1194/2401] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:55 2022] PUT /identity/v3/projects/ce0bcbe23a1d4854ad1565de299887c0/users/30aa83d0bc8c4b6ca1ea53ed6e9b8d4e/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 64 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:55.198360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:55.198799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:55.199180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:55.235859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] MFA Rules not processed for user `30aa83d0bc8c4b6ca1ea53ed6e9b8d4e`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:55.237822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:55.348984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:55.358207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1208/2402] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:55 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 161 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:55.371246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:55.373138 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:55.517504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5071062d-00e3-4bd7-8c58-a4d3be213885 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:55.523791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e85523ac61714842a523a22a7bc2888b', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:55.524598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:55.524991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:55.525392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:55.560088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:55.561566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:55.577061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1195/2403] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:55 2022] GET /identity/v3/auth/tokens => generated 3236 bytes in 206 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:06:57.862308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:57.868038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:57.987157 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e3e0d908-ff13-4891-aa70-b641d1e5fd34 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:57.993170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:57.993780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:57.994061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:57.994337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:57.996450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:57.999762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.001172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:58.012078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:58.018485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.020501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.033096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1209/2404] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:57 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:06:58.094029 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ce5d6f5c-8070-4b1d-9968-8a30b5ac89f3 service glance] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.100235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.101222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.101508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.101830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] PATH_INFO: `/v3/users/a59f5944a0db4710a43a6447019864d2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.103262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] RBAC: Authorizing `identity:delete_user(user_id=a59f5944a0db4710a43a6447019864d2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.105321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.156267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a59f5944a0db4710a43a6447019864d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.165048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a59f5944a0db4710a43a6447019864d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.165666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a59f5944a0db4710a43a6447019864d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.172660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a59f5944a0db4710a43a6447019864d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.184915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-475b581d-9b65-41f4-a19a-68d23b1023ba admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a59f5944a0db4710a43a6447019864d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.190786 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1196/2405] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:06:57 2022] DELETE /identity/v3/users/a59f5944a0db4710a43a6447019864d2 => generated 0 bytes in 192 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:06:58.380611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.396684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-62c72e63-cfa2-47d8-bf3c-e54eb2334815 service placement] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.471740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.472641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.473480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.473816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] PATH_INFO: `/v3/projects/644464d3831e468f8ab9957bb33817b0` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.475286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] RBAC: Authorizing `identity:delete_project(project_id=644464d3831e468f8ab9957bb33817b0)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.477377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.534064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '644464d3831e468f8ab9957bb33817b0', 'request_id': 'req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.545680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '644464d3831e468f8ab9957bb33817b0', 'request_id': 'req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.552166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '644464d3831e468f8ab9957bb33817b0', 'request_id': 'req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.563156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] The token cache is being invalidate because project 644464d3831e468f8ab9957bb33817b0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:06:58.563441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.564080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-053e7cbe-23e5-4c3f-b101-0ee7b6d9cc2d admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:06:58.565617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1210/2406] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:06:58 2022] DELETE /identity/v3/projects/644464d3831e468f8ab9957bb33817b0 => generated 0 bytes in 186 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:58.571314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.571608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.571890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.589557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:58.590758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:58.664608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1197/2407] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:58.673200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.673620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.673907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.696737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:58.697946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:58.705878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1211/2408] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:58 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 34 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:58.710867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.721085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c7a3861f-b79e-4dba-9554-85d94d2fdfc3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.726689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.727391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.727682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.727966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.728702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:06:58.729015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.730830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.743215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1198/2409] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:06:58 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:06:58.747663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.757120 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2ef3dd5-b1bb-41c4-a9af-fbf813d291e6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.761909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.762563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.762887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.763165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.763860 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.765776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.774854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:58.796177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1212/2410] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:06:58 2022] POST /identity/v3/projects => generated 371 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:58.800288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.810359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cad70a12-21b2-44fb-aa1f-e9f1b1ca99aa admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.819709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:06:58.870245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.870951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.871244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.871528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.872222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.874143 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.894174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1199/2411] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:06:58 2022] POST /identity/v3/users => generated 286 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:06:58.899037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.909761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3c9b65f-8b57-4ada-9df6-aed07dd599d6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.915297 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.915966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.916270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.916548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.917275 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.918941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.935366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1213/2412] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:06:58.936567 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.947908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-067995ce-e099-4bd4-9053-ac0832c78823 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.952610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.953270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.953569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.953849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:58.954574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:58.956211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b4716c3b-c0ea-4b0a-b89f-f7e79893b2d3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:58.967267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1200/2413] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:06:58 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:06:58.971744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:58.982683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-992c06a5-696c-444d-9548-211747aac67f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:58.987587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:58.988483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:58.988971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:58.989365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] PATH_INFO: `/v3/projects/74c0c8908908487bbf78fc18aca63c9b/users/a618f13fb1554a92b3c0a03eecf40a93/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:59.004886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] RBAC: Authorizing `identity:create_grant(project_id=74c0c8908908487bbf78fc18aca63c9b, user_id=a618f13fb1554a92b3c0a03eecf40a93, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:59.007669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:59.020806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1214/2414] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:06:58 2022] PUT /identity/v3/projects/74c0c8908908487bbf78fc18aca63c9b/users/a618f13fb1554a92b3c0a03eecf40a93/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:06:59.031209 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:59.031689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:59.031793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:59.058641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] MFA Rules not processed for user `a618f13fb1554a92b3c0a03eecf40a93`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:06:59.059899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:59.124616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1201/2415] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:06:59 2022] POST /identity/v3/auth/tokens => generated 3238 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:06:59.136244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:06:59.138515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:06:59.216611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aba91f5-4823-4179-bd93-7dac4c759c79 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:06:59.222220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:06:59.222930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:06:59.223288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:06:59.223493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:06:59.231872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:06:59.233486 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d808ac91-68bd-4fa2-823f-7cd79c3dba99 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:06:59.242559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1215/2416] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:06:59 2022] GET /identity/v3/auth/tokens => generated 3238 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:01.996347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:02.070397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-159009d5-c2dc-4514-a980-e53278801a24 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:02.075520 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:02.076404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:02.076693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:02.076977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] PATH_INFO: `/v3/users/a618f13fb1554a92b3c0a03eecf40a93` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:02.078718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a618f13fb1554a92b3c0a03eecf40a93)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:02.081047 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:02.120163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:INSERT=3,keystone:DELETE=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:02.154436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a618f13fb1554a92b3c0a03eecf40a93'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.161765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a618f13fb1554a92b3c0a03eecf40a93'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.162433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a618f13fb1554a92b3c0a03eecf40a93'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.173746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a618f13fb1554a92b3c0a03eecf40a93'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.188405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a618f13fb1554a92b3c0a03eecf40a93'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.195465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1202/2417] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:01 2022] DELETE /identity/v3/users/a618f13fb1554a92b3c0a03eecf40a93 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:02.200058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:02.200772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:02.200772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:02.220931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:02.222140 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:02.291878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1216/2418] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 93 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:02.302519 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:02.304100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:02.368862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41e73901-a37c-4782-96e0-e817b792b5e6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:02.372576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:02.373224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:02.373508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:02.373780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:02.383749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:02.385217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-18d77dad-adbe-4791-9c75-14a7800c4ecd service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:02.395260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1203/2419] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:02 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:02.681927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:02.683407 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:02.701469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e14647d2-562e-4936-b612-432e8db3ddb9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:02.706420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:02.707541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:02.707682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:02.707897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] PATH_INFO: `/v3/projects/74c0c8908908487bbf78fc18aca63c9b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:02.709696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=74c0c8908908487bbf78fc18aca63c9b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:02.721702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:02.873672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=2,keystone:DELETE=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:02.886521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '74c0c8908908487bbf78fc18aca63c9b', 'request_id': 'req-9878f87b-9587-4698-91fc-af1468712fd5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.893702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '74c0c8908908487bbf78fc18aca63c9b', 'request_id': 'req-9878f87b-9587-4698-91fc-af1468712fd5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.899691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '74c0c8908908487bbf78fc18aca63c9b', 'request_id': 'req-9878f87b-9587-4698-91fc-af1468712fd5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.911212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] The token cache is being invalidate because project 74c0c8908908487bbf78fc18aca63c9b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:02.911684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.912370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9878f87b-9587-4698-91fc-af1468712fd5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:02.914141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1217/2420] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:02 2022] DELETE /identity/v3/projects/74c0c8908908487bbf78fc18aca63c9b => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:02.919006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:02.919301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:02.919692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:02.938317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:02.939566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:03.002302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1204/2421] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:02 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:03.010953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.011251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.011537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.031253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:03.032502 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:03.041171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1218/2422] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:03.045630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.055403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9707ae77-25fe-4775-9376-c9c65d0fc22b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.059971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.060625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.060915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.061197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.061925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:03.062238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.063864 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.076110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1205/2423] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:03 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:03.080686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.091057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6582f05b-8ec4-4387-8236-f3d84cd44d51 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.095859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.096521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.096806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.097085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.097773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.100506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.125722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1219/2424] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/projects => generated 355 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:03.131854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.145401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9e9551c-34f3-41b1-9fd9-77e982e99d1f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.180322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:DELETE=6,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:03.221878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.223067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.223373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.223658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.225002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.227120 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.251806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1206/2425] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/users => generated 278 bytes in 120 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:03.256495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.277890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14db973e-c567-4190-a06f-9684f6d23234 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.282885 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.283611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.283902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.284178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.284907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.286656 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.297952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1220/2426] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:03 2022] GET /identity/v3/roles => generated 1420 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:03.303397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.313511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32c7a552-4b8e-464d-93aa-862977488e19 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.319086 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.319827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.320174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.320516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.321305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.322980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0dbdf292-cfa9-41a2-95fd-a9bb48025413 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.332850 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1207/2427] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:03 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:03.337709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.348088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cb46df-ca1d-4aca-ae63-044587394db1 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.353947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.355287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.355422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.355691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] PATH_INFO: `/v3/projects/05a3b3176f16448ca9d106b73df06b66/users/4c6abfc9e5ca40bd9aea39e2dcf7e4e9/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.373273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] RBAC: Authorizing `identity:create_grant(project_id=05a3b3176f16448ca9d106b73df06b66, user_id=4c6abfc9e5ca40bd9aea39e2dcf7e4e9, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.375971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.390564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1221/2428] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:03 2022] PUT /identity/v3/projects/05a3b3176f16448ca9d106b73df06b66/users/4c6abfc9e5ca40bd9aea39e2dcf7e4e9/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 53 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:03.403020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.403346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.403649 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.431726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] MFA Rules not processed for user `4c6abfc9e5ca40bd9aea39e2dcf7e4e9`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:03.432932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:03.500006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1208/2429] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/auth/tokens => generated 3222 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:03.504749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.574605 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c20aa235-520a-4576-b74c-43aa5645471d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.579701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.580509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.580797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.581078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.581786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.584597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.615023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1222/2430] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/projects => generated 357 bytes in 107 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:03.616328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.627456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59b10eeb-8e6e-441c-8e7d-31d6e2249627 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.695363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.696150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.696361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.696672 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.697413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.699501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.728515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1209/2431] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/users => generated 285 bytes in 110 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:03.729731 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.746846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4db4c079-d3e9-40d7-96b1-c7a08c532ad3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.752733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.753676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.753971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.754378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.756188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.766517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.788161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1223/2432] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:03 2022] GET /identity/v3/roles => generated 1420 bytes in 55 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:03.789346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.799846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e5fa85aa-b4ac-4c69-81a2-2ba2718b9594 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.804971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.805803 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.806151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.806514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] PATH_INFO: `/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd/users/13063a9d15a94eb78628a75d2a8129fa/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.825196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] RBAC: Authorizing `identity:create_grant(project_id=32bd0517ed9f4dd6a154164f751fa3bd, user_id=13063a9d15a94eb78628a75d2a8129fa, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.827915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.841364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1210/2433] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:03 2022] PUT /identity/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd/users/13063a9d15a94eb78628a75d2a8129fa/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:03.845450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.856195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73c751ae-3dcd-4ec0-961f-082f4c36b994 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.889558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=5,keystone:SELECT=91,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:03.923887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.924568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.924852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.925128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.926197 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.928552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6fe3ec62-7557-43a9-84db-ecc3e5a00e95 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.938822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1224/2434] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:03 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:03.944087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:03.950736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:03.955061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-350e8e89-092b-479e-af7c-ddd263262f3e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:03.960821 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:03.961712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.961999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.962279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] PATH_INFO: `/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd/users/13063a9d15a94eb78628a75d2a8129fa/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:03.964282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] RBAC: Authorizing `identity:create_grant(project_id=32bd0517ed9f4dd6a154164f751fa3bd, user_id=13063a9d15a94eb78628a75d2a8129fa, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:03.967030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:03.977737 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1211/2435] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:03 2022] PUT /identity/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd/users/13063a9d15a94eb78628a75d2a8129fa/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 35 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:03.987835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:03.988129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:03.988412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:04.015663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] MFA Rules not processed for user `13063a9d15a94eb78628a75d2a8129fa`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:04.016887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:04.087983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1225/2436] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:03 2022] POST /identity/v3/auth/tokens => generated 3291 bytes in 101 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:04.099835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:04.101775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:04.161938 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6867d956-2fc2-40d3-9ca7-c2ef88afa049 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:04.165689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:04.166336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:04.166625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:04.166944 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:04.176305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:04.178538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-28d56f81-a534-4efe-91a7-a1519058cc25 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:04.187877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1212/2437] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:04 2022] GET /identity/v3/auth/tokens => generated 3291 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:04.257389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:04.268027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0bd7949-dbff-434d-9b1e-d495f1fa8b50 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:04.272575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:04.273230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:04.273509 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:04.273784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:04.339335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:04.340931 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:04.351412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1226/2438] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:04 2022] GET /identity/v3/auth/tokens => generated 3222 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:06.703456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:06.703978 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:06.704087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:06.723545 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:06.729249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:06.790610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1213/2439] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:07:06 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:06.805563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:06.807412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:06.871741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:06.919548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1cd0d13-8f90-4aa7-b3b0-d5feffbd3f5f service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:06.926225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:06.927582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:06.927876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:06.928153 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:06.937477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:06.939335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b48e36ad-bd32-45f5-b2a7-bfbab36523ac service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:06.952609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1227/2440] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:07:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 147 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:07:16.783078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=54,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:16.948995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=8 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:19.461098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:19.525970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ee488070-d0bc-4839-a3ba-7a03d4727981 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:19.530491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:19.531397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:19.531679 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:19.531953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] PATH_INFO: `/v3/users/4c6abfc9e5ca40bd9aea39e2dcf7e4e9` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:19.533203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] RBAC: Authorizing `identity:delete_user(user_id=4c6abfc9e5ca40bd9aea39e2dcf7e4e9)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:19.535211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:19.584784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4c6abfc9e5ca40bd9aea39e2dcf7e4e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:19.591676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4c6abfc9e5ca40bd9aea39e2dcf7e4e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:19.592444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4c6abfc9e5ca40bd9aea39e2dcf7e4e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:19.600037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4c6abfc9e5ca40bd9aea39e2dcf7e4e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:19.612169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4c6abfc9e5ca40bd9aea39e2dcf7e4e9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:19.618148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1214/2441] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:19 2022] DELETE /identity/v3/users/4c6abfc9e5ca40bd9aea39e2dcf7e4e9 => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:19.623147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:19.623702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:19.623891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:19.642182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:19.643425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:19.704218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1228/2442] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:19.714524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:19.716132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:19.773888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-448e8a5c-eb78-4eb5-8b1f-8c171f51531f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:19.777578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:19.778219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:19.778502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:19.778789 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:19.788723 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:19.790333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:19.799853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1215/2443] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:19 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:20.075223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:20.076673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:20.094691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd4bffc6-0668-4655-8007-7a995247c237 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:20.099368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:20.100274 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.100421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.100988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] PATH_INFO: `/v3/projects/05a3b3176f16448ca9d106b73df06b66` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.102156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=05a3b3176f16448ca9d106b73df06b66)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:20.104122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:20.151043 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05a3b3176f16448ca9d106b73df06b66', 'request_id': 'req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.158442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05a3b3176f16448ca9d106b73df06b66', 'request_id': 'req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.164308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05a3b3176f16448ca9d106b73df06b66', 'request_id': 'req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.175978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] The token cache is being invalidate because project 05a3b3176f16448ca9d106b73df06b66 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:20.176268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.176956 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.178728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1229/2444] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:20 2022] DELETE /identity/v3/projects/05a3b3176f16448ca9d106b73df06b66 => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:20.183090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:20.193476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d2237a92-48bf-4b3d-9b27-6826cf8ad3f1 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:20.211705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:20.251087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:20.251958 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.252239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.252510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] PATH_INFO: `/v3/users/13063a9d15a94eb78628a75d2a8129fa` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.253717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] RBAC: Authorizing `identity:delete_user(user_id=13063a9d15a94eb78628a75d2a8129fa)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:20.256504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:20.306826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '13063a9d15a94eb78628a75d2a8129fa'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.319602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '13063a9d15a94eb78628a75d2a8129fa'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.320297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '13063a9d15a94eb78628a75d2a8129fa'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.328626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '13063a9d15a94eb78628a75d2a8129fa'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.341897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7d741a6a-bebd-4209-81ff-2a5580f2cfbc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '13063a9d15a94eb78628a75d2a8129fa'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.347989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1216/2445] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:20 2022] DELETE /identity/v3/users/13063a9d15a94eb78628a75d2a8129fa => generated 0 bytes in 165 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:20.650626 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:20.652148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:20.667844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f43a2a4b-ebee-4d41-9d03-70fee195cfd6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:20.729672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:20.730608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.730935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.731215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] PATH_INFO: `/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.732411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=32bd0517ed9f4dd6a154164f751fa3bd)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:20.734376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:20.764616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:20.783488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '32bd0517ed9f4dd6a154164f751fa3bd', 'request_id': 'req-106c2016-e769-473c-b38e-2584502d7cf7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.790675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '32bd0517ed9f4dd6a154164f751fa3bd', 'request_id': 'req-106c2016-e769-473c-b38e-2584502d7cf7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.796336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '32bd0517ed9f4dd6a154164f751fa3bd', 'request_id': 'req-106c2016-e769-473c-b38e-2584502d7cf7'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.808930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] The token cache is being invalidate because project 32bd0517ed9f4dd6a154164f751fa3bd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:20.809271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.809933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-106c2016-e769-473c-b38e-2584502d7cf7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:20.811596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1230/2446] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:20 2022] DELETE /identity/v3/projects/32bd0517ed9f4dd6a154164f751fa3bd => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:20.816488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.816972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.817072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.835857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:20.837045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:20.899849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1217/2447] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:20.908742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.909030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.909301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.929582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:20.930717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:20.938658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1231/2448] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:20 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:20.943822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:20.953705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43273b0f-946f-4668-80e1-df2505d92d68 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:20.958827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:20.959476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.959831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.960114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.960872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:20.961180 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:20.962800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:20.974220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1218/2449] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:20 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:20.978602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:20.988175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5c71220-7be1-4b98-bb9c-69132a1fea58 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:20.993511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:20.994160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:20.994449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:20.994727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:20.995450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:20.997352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.022909 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1232/2450] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:20 2022] POST /identity/v3/projects => generated 387 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:21.027080 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.037132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-654b421a-bc21-4312-b8e1-f235a5ada1c8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.063633 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:21.096559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.097224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.097506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.097779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.098451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.100745 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.120923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1219/2451] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:21 2022] POST /identity/v3/users => generated 294 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:21.125051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.135793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b2a27009-377b-476f-a1ad-f21d2bfc7898 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.140331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.140980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.141258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.141529 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.142236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.143855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.153985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1233/2452] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:21.157732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.167843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc9c53d7-2f42-4892-abfe-4e47ace50599 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.173188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.173844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.174135 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.174416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.175190 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.176779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a9762b96-dbbc-4666-b520-9e4224b522f0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.187454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1220/2453] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:21.191645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.202190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3e603ad-f8f0-4cee-a78d-a0088abf6cb9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.206686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.207588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.207866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.208145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] PATH_INFO: `/v3/projects/5daf1836480242ae809219e2408f407c/users/5df3ed148b0348259e5ac428832088c8/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.221934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] RBAC: Authorizing `identity:create_grant(project_id=5daf1836480242ae809219e2408f407c, user_id=5df3ed148b0348259e5ac428832088c8, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.224754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.237165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1234/2454] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:21 2022] PUT /identity/v3/projects/5daf1836480242ae809219e2408f407c/users/5df3ed148b0348259e5ac428832088c8/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 46 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:21.247999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.248295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.248577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.277491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] MFA Rules not processed for user `5df3ed148b0348259e5ac428832088c8`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:21.278759 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:21.342472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1221/2455] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:21 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:21.347107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.409364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db0f9f32-7d2d-45c1-b776-f080e01b7ce6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.415009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.415676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.415960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.416236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.416920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.418923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.448050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1235/2456] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:21 2022] POST /identity/v3/projects => generated 383 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:21.452160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.462797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-220b3019-3055-4b5f-8212-ab48dc34bd8f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.523564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.524240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.524528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.524810 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.525512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.527508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.548902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1222/2457] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:21 2022] POST /identity/v3/users => generated 298 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:21.553900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.564558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c54098d-4044-4718-a2d2-a609e8afa1a8 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.570997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.571672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.571952 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.572232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.572955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.576061 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.593225 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1236/2458] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:21.595188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.608136 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84c04e57-97a6-42ac-a75c-1eced70f0c0c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.620438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.621673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.622094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.622877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] PATH_INFO: `/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3/users/5123de8727964cd882da032bfc77fdae/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.649980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa3e061e688e46eb9091b9ee210f4fd3, user_id=5123de8727964cd882da032bfc77fdae, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.652825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.668268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1223/2459] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:21 2022] PUT /identity/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3/users/5123de8727964cd882da032bfc77fdae/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 74 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:21.672571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.685065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81e43a05-4d4a-487b-ad6d-7c4e9d579f13 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.722455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=6,keystone:SELECT=90,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:21.763695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.764383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.764666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.764942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.765679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.767351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56426798-aa7a-4c32-9a5d-7c6fc46a4ccc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.781747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1237/2460] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:21 2022] GET /identity/v3/roles => generated 1420 bytes in 106 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:21.783000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.788540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=93,keystone:INSERT=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:21.795370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5fe1e3c6-9614-45ac-8b17-5ba0fc04f5bb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:21.800514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:21.801438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.801756 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.802598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] PATH_INFO: `/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3/users/5123de8727964cd882da032bfc77fdae/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.806909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fa3e061e688e46eb9091b9ee210f4fd3, user_id=5123de8727964cd882da032bfc77fdae, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:21.810314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:21.820062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1224/2461] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:21 2022] PUT /identity/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3/users/5123de8727964cd882da032bfc77fdae/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:21.830235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:21.830525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:21.830833 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:21.858144 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] MFA Rules not processed for user `5123de8727964cd882da032bfc77fdae`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:21.859393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:21.933220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1238/2462] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:21 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:21.945483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:21.947248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.004724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-703004ea-30cb-4705-a210-ae85a5cd4937 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.008933 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.009571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.009859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.010144 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.019393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.020892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7689ef00-17df-4973-bf91-6420af0883d4 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.030271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1225/2463] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:21 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:22.059895 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:22.122958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e72d80b-d941-4d83-8606-f4c74404b23d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.127802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.128680 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.128967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.129248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] PATH_INFO: `/v3/users/5df3ed148b0348259e5ac428832088c8` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.130492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] RBAC: Authorizing `identity:delete_user(user_id=5df3ed148b0348259e5ac428832088c8)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.132534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.180293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5df3ed148b0348259e5ac428832088c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.182270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:DELETE=7 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:22.190481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5df3ed148b0348259e5ac428832088c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.191560 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5df3ed148b0348259e5ac428832088c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.199846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5df3ed148b0348259e5ac428832088c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.200497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5df3ed148b0348259e5ac428832088c8'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.220437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1239/2464] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:22 2022] DELETE /identity/v3/users/5df3ed148b0348259e5ac428832088c8 => generated 0 bytes in 161 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:22.225595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.225890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.226172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.244319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:22.245522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.303260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1226/2465] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:22.312891 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:22.314303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.372841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-effb21a3-92ce-4518-80d9-3f5672917eac admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.376548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.377193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.377481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.377759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.387235 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.388704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.398350 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1240/2466] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:22 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:22.666326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:22.667761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.680014 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-128fead6-4fa9-4c21-a4dc-daab36504541 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.685510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.686388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.686674 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.686989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] PATH_INFO: `/v3/projects/5daf1836480242ae809219e2408f407c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.688193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] RBAC: Authorizing `identity:delete_project(project_id=5daf1836480242ae809219e2408f407c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.690113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.738050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:22.739587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.741181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '5daf1836480242ae809219e2408f407c', 'request_id': 'req-fd037943-ef83-46ea-a7d8-899a09932189'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.748040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '5daf1836480242ae809219e2408f407c', 'request_id': 'req-fd037943-ef83-46ea-a7d8-899a09932189'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.756539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '5daf1836480242ae809219e2408f407c', 'request_id': 'req-fd037943-ef83-46ea-a7d8-899a09932189'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.759410 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21ab3e92-e5bb-42d6-a92d-5124e90b9079 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.768346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] The token cache is being invalidate because project 5daf1836480242ae809219e2408f407c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:22.768640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.769268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.771692 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1227/2467] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:22 2022] DELETE /identity/v3/projects/5daf1836480242ae809219e2408f407c => generated 0 bytes in 106 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:22.775420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:22.776744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.795005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:22.817752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.818929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.819218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.819493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] PATH_INFO: `/v3/users/30aa83d0bc8c4b6ca1ea53ed6e9b8d4e` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.820724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] RBAC: Authorizing `identity:delete_user(user_id=30aa83d0bc8c4b6ca1ea53ed6e9b8d4e)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.822683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.845470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd037943-ef83-46ea-a7d8-899a09932189 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:22.850739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:22.851650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.851932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.852277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] PATH_INFO: `/v3/users/5123de8727964cd882da032bfc77fdae` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.853433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5123de8727964cd882da032bfc77fdae)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:22.855449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:22.872985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '30aa83d0bc8c4b6ca1ea53ed6e9b8d4e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.881412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '30aa83d0bc8c4b6ca1ea53ed6e9b8d4e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.882171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '30aa83d0bc8c4b6ca1ea53ed6e9b8d4e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.889960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '30aa83d0bc8c4b6ca1ea53ed6e9b8d4e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.890593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f83184ea-6f2d-46cd-8a86-127fdb3bf734 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '30aa83d0bc8c4b6ca1ea53ed6e9b8d4e'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.896925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:INSERT=2,keystone:DELETE=12,keystone:SELECT=86 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:22.903981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5123de8727964cd882da032bfc77fdae'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.911079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1241/2468] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:22 2022] DELETE /identity/v3/users/30aa83d0bc8c4b6ca1ea53ed6e9b8d4e => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:22.911336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5123de8727964cd882da032bfc77fdae'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.911905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5123de8727964cd882da032bfc77fdae'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.915789 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:22.916293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:22.916876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:22.919013 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5123de8727964cd882da032bfc77fdae'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.933940 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5123de8727964cd882da032bfc77fdae'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:22.935341 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:22.936771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:22.943422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1228/2469] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:22 2022] DELETE /identity/v3/users/5123de8727964cd882da032bfc77fdae => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:23.011787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1242/2470] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:22 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:23.023907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.025602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.035248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.037582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.094402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-620e69d7-8dd1-45da-86d2-f752980a68e5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.100439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.101446 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.101741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.102022 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.105207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78cacbd2-b59d-4849-acfd-0ae55b98f18f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.110470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.111239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.111641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.111980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.116083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.154788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:23.171750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.173210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.183880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1229/2471] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 161 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:23.194466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.196070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedabaa1-beb8-4faf-867d-21eadf0bfad6 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.206060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1243/2472] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:23 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 171 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:23.434432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.435976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.449262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acac22bf-7825-498e-884c-37a1eea6e0c3 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.454031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.454941 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.455231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.455512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] PATH_INFO: `/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.456728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fa3e061e688e46eb9091b9ee210f4fd3)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.459090 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.518084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fa3e061e688e46eb9091b9ee210f4fd3', 'request_id': 'req-8d9081f5-f606-4512-bf17-2e1b0e002272'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.524260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fa3e061e688e46eb9091b9ee210f4fd3', 'request_id': 'req-8d9081f5-f606-4512-bf17-2e1b0e002272'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.531870 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fa3e061e688e46eb9091b9ee210f4fd3', 'request_id': 'req-8d9081f5-f606-4512-bf17-2e1b0e002272'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.543984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] The token cache is being invalidate because project fa3e061e688e46eb9091b9ee210f4fd3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:23.544277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.544928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-8d9081f5-f606-4512-bf17-2e1b0e002272 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.546500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1230/2473] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:23 2022] DELETE /identity/v3/projects/fa3e061e688e46eb9091b9ee210f4fd3 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:23.551612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.552478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.552772 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.572765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:23.574004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.619143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:DELETE=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:23.654417 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1244/2474] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:23.663905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.664213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.664487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.690851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:23.691997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.700979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1231/2475] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:23 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:23.705455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.712620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.714320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.717726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0cffa35d-b852-43b9-aaf2-752923e29ca1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.730287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.731522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.731875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.732160 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.733241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:23.733554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.733766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a6f0077-a4de-4df5-a10b-760ed6e45faa None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.735919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.738695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.740299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.740636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.740851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] PATH_INFO: `/v3/projects/ce0bcbe23a1d4854ad1565de299887c0` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.742109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ce0bcbe23a1d4854ad1565de299887c0)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.745327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.759240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1245/2476] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:23 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 54 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:23.762886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.773155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-32da47a2-ad4a-403b-a904-d0e15f54cda6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.778432 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.779192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.779519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.779865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.780649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.782572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.807075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ce0bcbe23a1d4854ad1565de299887c0', 'request_id': 'req-136a56d0-68ac-4099-a297-ddaef21208d6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.813453 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1246/2477] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:23 2022] POST /identity/v3/projects => generated 387 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:23.817290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ce0bcbe23a1d4854ad1565de299887c0', 'request_id': 'req-136a56d0-68ac-4099-a297-ddaef21208d6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.817525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.825784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ce0bcbe23a1d4854ad1565de299887c0', 'request_id': 'req-136a56d0-68ac-4099-a297-ddaef21208d6'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.837547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] The token cache is being invalidate because project ce0bcbe23a1d4854ad1565de299887c0 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:23.837849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.838515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:23.840125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1232/2478] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:23 2022] DELETE /identity/v3/projects/ce0bcbe23a1d4854ad1565de299887c0 => generated 0 bytes in 129 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:23.897141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad1b7b74-3081-47bc-93f4-62dfb989e4e4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:23.902330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:23.903141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:23.903493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:23.903836 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:23.904645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:23.906743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:23.929393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1247/2479] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:23 2022] POST /identity/v3/users => generated 294 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:23.933578 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:23.935766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:23.993743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:24.017349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-136a56d0-68ac-4099-a297-ddaef21208d6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.022675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.023799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.023799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.024010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.024757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.026398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.037616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1233/2480] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:23 2022] GET /identity/v3/roles => generated 1420 bytes in 105 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:24.041960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.056164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-61b9168a-0ccd-44c5-aa31-453cf6346378 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.061362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.062048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.062327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.062597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.063642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.065316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0ec6e079-33cd-4d46-a614-caf1d8ab28b1 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.077438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1248/2481] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:24 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:24.081518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.092322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-730b9b08-084c-47ea-b363-cb57d21b1e4d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.097231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.098148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.098437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.098755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] PATH_INFO: `/v3/projects/8afa24eea83e4843bf086fcdb1cf768f/users/b5c5a1b821b144c99b29812b6745dd2f/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.116023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8afa24eea83e4843bf086fcdb1cf768f, user_id=b5c5a1b821b144c99b29812b6745dd2f, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.118739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.133071 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1234/2482] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:24 2022] PUT /identity/v3/projects/8afa24eea83e4843bf086fcdb1cf768f/users/b5c5a1b821b144c99b29812b6745dd2f/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:24.143133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.143426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.143707 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.170660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] MFA Rules not processed for user `b5c5a1b821b144c99b29812b6745dd2f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:24.171970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:24.239970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1249/2483] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:24 2022] POST /identity/v3/auth/tokens => generated 3254 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:24.244589 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.316829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1e0e6be-4bfe-43c3-9f4f-ab6a61b3b065 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.322696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.323284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.323637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.323991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.324774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.326833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.358259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1235/2484] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:24 2022] POST /identity/v3/projects => generated 387 bytes in 114 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:24.362313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.374869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9dfb3ecd-adb6-445d-8cb2-261be4463012 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.384914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:24.450570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.451454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.451668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.451936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.452654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.454648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.478352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1250/2485] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:24 2022] POST /identity/v3/users => generated 300 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:24.481966 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.493228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3336aac8-e09f-45b9-8cdf-ccf5d9231453 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.497913 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.498591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.498927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.499203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.499932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.501554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.513695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1236/2486] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:24 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:24.517972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.529285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-322e24a3-eb29-4212-850c-aaf6641403cb admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.534135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.535092 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.535371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.535650 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] PATH_INFO: `/v3/projects/fe0f52b1455a41eb9fa4bba088663dda/users/ccc72a68153a417d944737ef0ca18a4d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.551936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe0f52b1455a41eb9fa4bba088663dda, user_id=ccc72a68153a417d944737ef0ca18a4d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.554888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.573054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1251/2487] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:24 2022] PUT /identity/v3/projects/fe0f52b1455a41eb9fa4bba088663dda/users/ccc72a68153a417d944737ef0ca18a4d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 56 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:24.577287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.588985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e95dee12-e8cb-445d-9406-e00a9f165d7f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.652823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.653542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.653833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.654118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.654872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.656488 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5fd3b12-0a39-40d1-b840-787b68a7bc32 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.667898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1237/2488] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:24 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:24.672252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.683037 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-525add60-967d-485f-98ba-3445cd55bc28 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.687669 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.688564 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.688843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.689123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] PATH_INFO: `/v3/projects/fe0f52b1455a41eb9fa4bba088663dda/users/ccc72a68153a417d944737ef0ca18a4d/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.691209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] RBAC: Authorizing `identity:create_grant(project_id=fe0f52b1455a41eb9fa4bba088663dda, user_id=ccc72a68153a417d944737ef0ca18a4d, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.694722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.706016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1252/2489] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:24 2022] PUT /identity/v3/projects/fe0f52b1455a41eb9fa4bba088663dda/users/ccc72a68153a417d944737ef0ca18a4d/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:24.717397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.717682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.717957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.744412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] MFA Rules not processed for user `ccc72a68153a417d944737ef0ca18a4d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:24.745573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:24.758442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:24.818917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1238/2490] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:24 2022] POST /identity/v3/auth/tokens => generated 3321 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:24.828057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:24.829743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:24.889536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9c20d07d-a403-4fa3-b9dc-94826604c55b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:24.893162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:24.893803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:24.894080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:24.894384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:24.903771 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:24.905198 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d9cd8de8-f2f2-49c5-bc5b-cb03dc6e8afa service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:24.915354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1253/2491] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:24 2022] GET /identity/v3/auth/tokens => generated 3321 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:24.981267 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:25.055028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b950fd0d-c955-4886-bf3a-5f2db2b8c37a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:25.060156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:25.061116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:25.061474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:25.062077 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] PATH_INFO: `/v3/users/b5c5a1b821b144c99b29812b6745dd2f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:25.063396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] RBAC: Authorizing `identity:delete_user(user_id=b5c5a1b821b144c99b29812b6745dd2f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:25.065471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:25.122391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b5c5a1b821b144c99b29812b6745dd2f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:25.130875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b5c5a1b821b144c99b29812b6745dd2f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:25.131693 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b5c5a1b821b144c99b29812b6745dd2f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:25.142405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b5c5a1b821b144c99b29812b6745dd2f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:25.156030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b5c5a1b821b144c99b29812b6745dd2f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:25.163706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1239/2492] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:24 2022] DELETE /identity/v3/users/b5c5a1b821b144c99b29812b6745dd2f => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:25.168381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:25.168381 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:25.168647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:25.186712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:25.188145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:25.258554 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1254/2493] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:25 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:25.269985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:25.272587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:25.333536 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:25.364272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f159087-01db-47d4-b406-43b79094d149 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:25.368965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:25.369719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:25.370099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:25.370470 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:25.382562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:25.384178 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:25.402328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1240/2494] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:25 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:25.896695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:25.898285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:25.929644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97c304e9-4e62-42ed-9a18-d8f1e54a520e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:25.934916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:25.935821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:25.936108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:25.936390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] PATH_INFO: `/v3/projects/8afa24eea83e4843bf086fcdb1cf768f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:25.937806 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8afa24eea83e4843bf086fcdb1cf768f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:25.939844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:25.941684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:26.019419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8afa24eea83e4843bf086fcdb1cf768f', 'request_id': 'req-b35cef24-a07b-4765-8c77-18f612ee4590'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.027421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8afa24eea83e4843bf086fcdb1cf768f', 'request_id': 'req-b35cef24-a07b-4765-8c77-18f612ee4590'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.033511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8afa24eea83e4843bf086fcdb1cf768f', 'request_id': 'req-b35cef24-a07b-4765-8c77-18f612ee4590'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.045748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] The token cache is being invalidate because project 8afa24eea83e4843bf086fcdb1cf768f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:26.046162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.046884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.048651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1255/2495] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:25 2022] DELETE /identity/v3/projects/8afa24eea83e4843bf086fcdb1cf768f => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:26.053053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:26.062617 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a48ef21-9acb-4536-a164-0cfabd765bd8 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:26.122137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:26.123076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.123363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.123639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] PATH_INFO: `/v3/users/ccc72a68153a417d944737ef0ca18a4d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.124838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ccc72a68153a417d944737ef0ca18a4d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:26.126830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:26.181303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ccc72a68153a417d944737ef0ca18a4d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.191075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ccc72a68153a417d944737ef0ca18a4d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.193566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ccc72a68153a417d944737ef0ca18a4d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.203223 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ccc72a68153a417d944737ef0ca18a4d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.218042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ccc72a68153a417d944737ef0ca18a4d'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.224865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1241/2496] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:26 2022] DELETE /identity/v3/users/ccc72a68153a417d944737ef0ca18a4d => generated 0 bytes in 173 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:26.566492 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:26.568558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:26.595777 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b35cef24-a07b-4765-8c77-18f612ee4590 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:26.678540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:26.679444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.679820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.680574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] PATH_INFO: `/v3/projects/fe0f52b1455a41eb9fa4bba088663dda` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.681627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] RBAC: Authorizing `identity:delete_project(project_id=fe0f52b1455a41eb9fa4bba088663dda)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:26.686070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:26.769582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'fe0f52b1455a41eb9fa4bba088663dda', 'request_id': 'req-f7a21451-f685-43c7-8357-8cd03e67f571'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.780497 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'fe0f52b1455a41eb9fa4bba088663dda', 'request_id': 'req-f7a21451-f685-43c7-8357-8cd03e67f571'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.791421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'fe0f52b1455a41eb9fa4bba088663dda', 'request_id': 'req-f7a21451-f685-43c7-8357-8cd03e67f571'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.797797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:26.799321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:26.810404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] The token cache is being invalidate because project fe0f52b1455a41eb9fa4bba088663dda was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:26.811091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.812281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f7a21451-f685-43c7-8357-8cd03e67f571 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.812749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e56a514e-fcdc-462a-ba7f-3307642ac8a8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:26.813989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1256/2497] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:26 2022] DELETE /identity/v3/projects/fe0f52b1455a41eb9fa4bba088663dda => generated 0 bytes in 248 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:26.819480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d139fcaf-49c1-46a7-b490-adbca54c4da3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.819785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d139fcaf-49c1-46a7-b490-adbca54c4da3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.820049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d139fcaf-49c1-46a7-b490-adbca54c4da3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.839726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d139fcaf-49c1-46a7-b490-adbca54c4da3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:26.841694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d139fcaf-49c1-46a7-b490-adbca54c4da3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:26.856062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:26.870228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:26.883104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:26.883997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.884291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.884574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] PATH_INFO: `/v3/users/8f471531dac249d994f864102f7b9d3f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.885773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8f471531dac249d994f864102f7b9d3f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:26.887762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:26.910694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1257/2498] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 92 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:26.919640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.919977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.920360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.939545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:26.940799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:26.943709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8f471531dac249d994f864102f7b9d3f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.950206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1258/2499] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:26.951569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8f471531dac249d994f864102f7b9d3f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.952227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8f471531dac249d994f864102f7b9d3f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.954450 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:26.960125 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8f471531dac249d994f864102f7b9d3f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.964408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd23b459-808f-494c-8918-8b1e913b6260 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:26.969110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:26.969754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:26.970049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:26.970328 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:26.971099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:26.971448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:26.972107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8f471531dac249d994f864102f7b9d3f'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:26.973016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:26.978198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1242/2500] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:26 2022] DELETE /identity/v3/users/8f471531dac249d994f864102f7b9d3f => generated 0 bytes in 181 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:26.985519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1259/2501] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:26 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:07:26.990201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:26.991871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.068881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ebd24b8d-3fdb-408d-af9f-0cb800c36100 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.074341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.075051 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.075344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.075624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.076363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.078389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.105549 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1243/2502] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:26 2022] POST /identity/v3/projects => generated 361 bytes in 116 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:27.109850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.120214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c71cc42-5423-43f2-89c6-655cff28e037 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.158171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.160174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.188738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.189553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.189964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.190352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.191478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.194379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.215429 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1260/2503] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/users => generated 281 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:27.219531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.230207 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5938d13-e581-449c-baa8-59bb7c280986 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.235009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.235655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.235933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.236205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.236949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.238551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.238973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=88,keystone:DELETE=10,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:27.243829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97d2c7d8-736d-48ac-91be-37d2b0bb5dc7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.249749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1261/2504] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:27.250725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.252163 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.252531 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.252892 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] PATH_INFO: `/v3/projects/3c68e39bc1724fda980be3463bbf7d7b` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.253958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.254748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c68e39bc1724fda980be3463bbf7d7b)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.257427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.264100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b3d7a767-e50a-445f-b552-c98eebd572c4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.269238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.270047 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.270617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.270936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.271829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.273588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.284639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1262/2505] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:27.289748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.301030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1bdac85f-16bb-4861-9024-aae0b3d39eab admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.320279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c68e39bc1724fda980be3463bbf7d7b', 'request_id': 'req-f890d1d3-4020-4177-942a-0588156b0ec1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.327755 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c68e39bc1724fda980be3463bbf7d7b', 'request_id': 'req-f890d1d3-4020-4177-942a-0588156b0ec1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.338171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c68e39bc1724fda980be3463bbf7d7b', 'request_id': 'req-f890d1d3-4020-4177-942a-0588156b0ec1'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.356539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] The token cache is being invalidate because project 3c68e39bc1724fda980be3463bbf7d7b was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:27.356956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.357763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f890d1d3-4020-4177-942a-0588156b0ec1 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.359500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1244/2506] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:27 2022] DELETE /identity/v3/projects/3c68e39bc1724fda980be3463bbf7d7b => generated 0 bytes in 202 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:27.364971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a8e8c-06a5-478d-8b44-fdf35151e4e3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.365532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a8e8c-06a5-478d-8b44-fdf35151e4e3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.366038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-432a8e8c-06a5-478d-8b44-fdf35151e4e3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.367873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.368636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.368934 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.369164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] PATH_INFO: `/v3/projects/ca2d192fa5ee4c68b154fdbd1f4b0f64/users/7b86d801b632447c8116bb710706dd69/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.382978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:27.388646 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-432a8e8c-06a5-478d-8b44-fdf35151e4e3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:27.388997 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] RBAC: Authorizing `identity:create_grant(project_id=ca2d192fa5ee4c68b154fdbd1f4b0f64, user_id=7b86d801b632447c8116bb710706dd69, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.390608 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-432a8e8c-06a5-478d-8b44-fdf35151e4e3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.392058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-98334c4e-f58d-4c14-9133-52c04593ebbf admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.401868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1245/2507] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 38 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:27.409461 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.409670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1263/2508] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:27 2022] PUT /identity/v3/projects/ca2d192fa5ee4c68b154fdbd1f4b0f64/users/7b86d801b632447c8116bb710706dd69/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 120 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:27.409774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.410102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.421219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.421592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.421946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.438594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:27.439881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.452722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] MFA Rules not processed for user `7b86d801b632447c8116bb710706dd69`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:27.453926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.502226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1246/2509] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:27.506689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.515377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4853138e-6243-4760-9665-2a5879ec6951 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.520057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1264/2510] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3228 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:27.520263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.520775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.521059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.521445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.522170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:27.522478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.524703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.538593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1247/2511] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:07:27.542794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.555000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.555505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dc9f4868-3962-44a7-83e7-5a33d7ee7fbe None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.556704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.560141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.560827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.561106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.561374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.562057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.564020 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.591191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1265/2512] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/projects => generated 367 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:27.595408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.620896 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634b3247-ef42-4e6e-875d-d7336176d8a5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.643007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:27.664863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ccbb2bb-bc86-49f9-a178-751d426f7909 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.669531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.670202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.670487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.670763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.671507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.673446 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.682418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.683101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.683381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.683657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.695469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1266/2513] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/users => generated 284 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:27.699556 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.711126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7aa03114-2771-40fc-ae67-5c282397879f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.716021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.716676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.717024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.717308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.718062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.719701 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.730686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1267/2514] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:27.734687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.743829 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.745183 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-843ad799-e46f-451c-a37b-e67c17eb70ed admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.745548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.749705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.750802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.750802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.751032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.751602 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.753173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.756148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1248/2515] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/auth/tokens => generated 3228 bytes in 202 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:07:27.768928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1268/2516] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:27.770525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.772122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.772492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.782099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-02d3e361-6664-44e8-b8bd-80c089902ee7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.786076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2643799-ef14-4203-9f4c-508562450fd1 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:27.786829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.787709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.787994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.788288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] PATH_INFO: `/v3/projects/bc25de90dd7b432f80eeb57d4ac3730f/users/c269e3e90a274aa5a3384339960dad33/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.792348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:27.793373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.793655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.793931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] PATH_INFO: `/v3/users/7b86d801b632447c8116bb710706dd69` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.795215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7b86d801b632447c8116bb710706dd69)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.797591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.803645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] RBAC: Authorizing `identity:create_grant(project_id=bc25de90dd7b432f80eeb57d4ac3730f, user_id=c269e3e90a274aa5a3384339960dad33, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:27.806871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a664155e-5992-4c47-a81d-08109e3856de admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:27.823375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1269/2517] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:27 2022] PUT /identity/v3/projects/bc25de90dd7b432f80eeb57d4ac3730f/users/c269e3e90a274aa5a3384339960dad33/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 52 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:27.835800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.836088 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.836365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.842155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:27.847976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7b86d801b632447c8116bb710706dd69'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.855778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7b86d801b632447c8116bb710706dd69'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.856399 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7b86d801b632447c8116bb710706dd69'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.864631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7b86d801b632447c8116bb710706dd69'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.865524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] MFA Rules not processed for user `c269e3e90a274aa5a3384339960dad33`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:27.866904 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.877535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-2ffb42a3-ef1d-4da0-978d-b24166fd8e78 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7b86d801b632447c8116bb710706dd69'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:27.885460 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1249/2518] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:27 2022] DELETE /identity/v3/users/7b86d801b632447c8116bb710706dd69 => generated 0 bytes in 115 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:27.890249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:27.890538 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:27.890842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:27.907434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:27.908634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.933471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1270/2519] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3234 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:27.945371 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.946751 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:27.956500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:27.973366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1250/2520] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:27.983741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:27.985208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:28.003941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2eef07ae-74b3-4bbe-9ee3-86915269ad7f None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.045376 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0adaa0af-d40a-4989-8dd2-fb33a44bb12d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.049043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.049683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.049968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.050243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.054029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.054678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.055015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.055299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.060364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.061799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.072694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1251/2521] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:07:28.117404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.119182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2949183d-ee99-4311-b082-39a311c029e9 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.131886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1271/2522] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:27 2022] GET /identity/v3/auth/tokens => generated 3234 bytes in 187 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:07:28.355194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.356839 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:28.422408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44cf1723-eab6-4dfd-81c1-12cd9c796075 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.437797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.439145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.439435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.439722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] PATH_INFO: `/v3/projects/ca2d192fa5ee4c68b154fdbd1f4b0f64` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.441024 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=ca2d192fa5ee4c68b154fdbd1f4b0f64)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.444238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.513959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ca2d192fa5ee4c68b154fdbd1f4b0f64', 'request_id': 'req-0e9fa041-21f5-44fd-b857-284dd4893fb9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:28.521156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ca2d192fa5ee4c68b154fdbd1f4b0f64', 'request_id': 'req-0e9fa041-21f5-44fd-b857-284dd4893fb9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:28.529111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ca2d192fa5ee4c68b154fdbd1f4b0f64', 'request_id': 'req-0e9fa041-21f5-44fd-b857-284dd4893fb9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:28.540985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] The token cache is being invalidate because project ca2d192fa5ee4c68b154fdbd1f4b0f64 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:28.541278 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:28.541912 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0e9fa041-21f5-44fd-b857-284dd4893fb9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:28.543854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1252/2523] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:28 2022] DELETE /identity/v3/projects/ca2d192fa5ee4c68b154fdbd1f4b0f64 => generated 0 bytes in 189 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:28.548712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.549007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.549288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.556249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:28.569566 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:28.570953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:28.637270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1272/2524] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:28.645101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.645368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.645654 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.665648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:28.666902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:28.675110 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1253/2525] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:28.680058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.689888 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f606a85-c0cd-484b-bb85-028e999a49e0 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.695525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.696190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.696593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.696880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.697887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:28.698204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.699855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.712875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1273/2526] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:28 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:28.717327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.726590 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43e5a74d-9dff-4c95-97a0-0398547cd614 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.731455 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.732111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.732383 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.732662 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.733362 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.735304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.762279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1254/2527] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:28 2022] POST /identity/v3/projects => generated 373 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:28.766325 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.777244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e7b10e-3bfe-4acf-9da6-32771134213e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.834854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.835526 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.835812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.836089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.836915 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.838924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.858167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1274/2528] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:28 2022] POST /identity/v3/users => generated 287 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:28.862539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.874322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2b509d-88f6-454b-ab0d-8662cb82bf56 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.879007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.879712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.879986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.880263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.881026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.882796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.893573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1255/2529] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:28.897544 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.907747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6042d018-2b00-4fb5-af7b-62067b06ce02 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.913026 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.913677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.913960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.914237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.914998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.916601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8fc6c087-8ccd-485f-bb11-7d04fe9f262e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.926456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1275/2530] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:28 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:28.931238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:28.941065 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fcffad0-3fcd-4127-8b65-f13312eeb170 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:28.945995 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:28.946909 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.947196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.947472 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] PATH_INFO: `/v3/projects/54b31a874a6b4ec48f9a938d817c4637/users/ac64444a4b794026a5e95584d31f1c04/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:28.961736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] RBAC: Authorizing `identity:create_grant(project_id=54b31a874a6b4ec48f9a938d817c4637, user_id=ac64444a4b794026a5e95584d31f1c04, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:28.964442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:28.977559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1256/2531] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:28 2022] PUT /identity/v3/projects/54b31a874a6b4ec48f9a938d817c4637/users/ac64444a4b794026a5e95584d31f1c04/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:28.982598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e4a01-7d97-4509-88ee-5490ec353129 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:28.983182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e4a01-7d97-4509-88ee-5490ec353129 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:28.983182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7e4a01-7d97-4509-88ee-5490ec353129 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:29.002364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cb7e4a01-7d97-4509-88ee-5490ec353129 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:29.003582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cb7e4a01-7d97-4509-88ee-5490ec353129 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:29.061855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1276/2532] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:28 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:29.073497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:29.075247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:29.090132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:29.134853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d0f7363-e1cd-4c12-af56-4cd5d24759f1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:29.138510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:29.139195 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:29.139484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:29.139769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:29.148856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:29.150227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:29.159624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1257/2533] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:29 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:07:29.773203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb745e0-a6ca-4368-9f50-0e7d12ed57ad None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:29.773757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb745e0-a6ca-4368-9f50-0e7d12ed57ad None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:29.774057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cdb745e0-a6ca-4368-9f50-0e7d12ed57ad None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:29.793685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:29.820716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cdb745e0-a6ca-4368-9f50-0e7d12ed57ad None None] MFA Rules not processed for user `ac64444a4b794026a5e95584d31f1c04`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:29.823052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cdb745e0-a6ca-4368-9f50-0e7d12ed57ad None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:29.903783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1277/2534] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:29 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 132 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:29.915360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:29.916782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:29.977601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-13ad6245-eb42-4454-aeda-e7590420dea6 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:29.981710 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:29.982621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:29.982985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:29.983307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:29.993539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:29.995004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:30.004282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1258/2535] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:29 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 89 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:31.341978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:31.342481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:31.342591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:31.360200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] MFA Rules not processed for user `ac64444a4b794026a5e95584d31f1c04`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:31.361477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:31.369082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1278/2536] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:31 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:31.597678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:31.600044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:31.612236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b064c41-0fe1-41f8-991c-c91507b95b5c service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:31.615925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:31.616562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:31.616844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:31.617124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:31.618883 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:31.628257 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:31.629886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-634fda57-a51e-4663-bb55-4bb6e3a1c3cb service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:31.637988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1259/2537] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:31 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:36.518262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:36.519655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:36.533988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d33ad114-d62d-4521-b178-15fcac625905 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:36.538460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:36.539362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:36.539645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:36.539919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] PATH_INFO: `/v3/users/c269e3e90a274aa5a3384339960dad33` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:36.541094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c269e3e90a274aa5a3384339960dad33)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:36.543077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:36.589148 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c269e3e90a274aa5a3384339960dad33'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:36.596264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c269e3e90a274aa5a3384339960dad33'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:36.596879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c269e3e90a274aa5a3384339960dad33'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:36.604099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c269e3e90a274aa5a3384339960dad33'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:36.604993 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c269e3e90a274aa5a3384339960dad33'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:36.622358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1279/2538] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:36 2022] DELETE /identity/v3/users/c269e3e90a274aa5a3384339960dad33 => generated 0 bytes in 105 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:36.626927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:36.627402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:36.627529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:36.645432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:36.647431 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:36.705956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1260/2539] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:36.715525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:36.717079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:36.776089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2a4e192e-f43f-4d5b-a378-887028c313e6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:36.779867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:36.780518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:36.780804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:36.781081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:36.791025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:36.792597 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4775fcac-3570-4634-82e3-e3edf1df4a86 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:36.802292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1280/2540] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:36 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:37.079857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.081359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:37.096123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b42f1cc3-fc76-40ef-9cc9-3135a12f046d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.101729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.102602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.102917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.103192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] PATH_INFO: `/v3/projects/bc25de90dd7b432f80eeb57d4ac3730f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.104398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=bc25de90dd7b432f80eeb57d4ac3730f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.106344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.148906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:DELETE=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:37.155955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'bc25de90dd7b432f80eeb57d4ac3730f', 'request_id': 'req-36ddcfae-5398-4910-8d54-5768ad3ac7d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:37.161427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'bc25de90dd7b432f80eeb57d4ac3730f', 'request_id': 'req-36ddcfae-5398-4910-8d54-5768ad3ac7d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:37.168888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'bc25de90dd7b432f80eeb57d4ac3730f', 'request_id': 'req-36ddcfae-5398-4910-8d54-5768ad3ac7d2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:37.180500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] The token cache is being invalidate because project bc25de90dd7b432f80eeb57d4ac3730f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:37.180783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:37.181410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-36ddcfae-5398-4910-8d54-5768ad3ac7d2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:37.183092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1261/2541] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:37 2022] DELETE /identity/v3/projects/bc25de90dd7b432f80eeb57d4ac3730f => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:37.187903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.188191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.188463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.200724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:37.207074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:37.208842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:37.268084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1281/2542] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:37.276791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.277121 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.277420 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.296587 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:37.297776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:37.306541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1262/2543] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:37.310714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.320796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb355871-31f1-4b14-8023-4edf4dffe255 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.325385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.326065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.326346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.326621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.327382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:37.327687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.329287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.340770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1282/2544] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:37 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:37.344930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.354487 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-147fcc41-4af7-4caa-a79a-aa0d2a12c815 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.359131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.359773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.360054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.360327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.361021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.362942 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.392574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1263/2545] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/projects => generated 405 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:37.393793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.404491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-45363dc6-d6ed-4da3-bb3a-e7a442fbac39 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.462311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.462958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.463241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.463513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.464190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.466105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.486008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1283/2546] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/users => generated 303 bytes in 93 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:37.490456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.500537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aded9d7e-99a1-4394-aa65-0426ec72cfcd admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.505126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.505904 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.506621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.506621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.507204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.508814 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.519139 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1264/2547] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:37.523681 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.533712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-76ce04d1-7999-4891-a31a-2c9a79ca22c9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.539029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.539677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.539964 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.540240 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.540954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.542582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84fda11e-0b77-407c-8d74-0cfed9b39b75 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.552364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1284/2548] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:37 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:37.556417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.566293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a086cf74-ecae-4539-9871-26cc72701063 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.571248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.572118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.572404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.572683 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] PATH_INFO: `/v3/projects/a4e01a7e63cf46f08ecb2a90a71da826/users/7a52eb7ff3bc46bb84b1ad364a718889/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.587553 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a4e01a7e63cf46f08ecb2a90a71da826, user_id=7a52eb7ff3bc46bb84b1ad364a718889, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.590235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.603551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1265/2549] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:37 2022] PUT /identity/v3/projects/a4e01a7e63cf46f08ecb2a90a71da826/users/7a52eb7ff3bc46bb84b1ad364a718889/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:37.613886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.614200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.614481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.640320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] MFA Rules not processed for user `7a52eb7ff3bc46bb84b1ad364a718889`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:37.641522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:37.707049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1285/2550] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:37.711888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.773680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2b2a752-e18e-4285-b27b-a93fd806af08 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.778732 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.779570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.780004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.780335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.781296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.783719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.810989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1266/2551] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/projects => generated 405 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:37.815048 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.821296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:37.827648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c90f1847-b639-48d7-bc40-3690654ee1e8 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.884531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.885249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.885530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.885805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.886506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.888595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.910443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1286/2552] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:37 2022] POST /identity/v3/users => generated 309 bytes in 96 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:37.914963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.930546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-595cada3-2678-44e3-b760-f7b9cc8c18d0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.936358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.937043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.937329 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.937638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.938334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.939969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:37.952983 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1267/2553] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:37 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:37.957182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:37.969831 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-780d8dd0-0aed-4f0c-b02d-954712e107d5 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:37.974255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:37.975774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:37.976125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:37.976471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] PATH_INFO: `/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d/users/7a58e4daf6024eb69fbae61e71e71e65/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:37.995272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c01bb6db1ec4ff3ab9423ccc9f5bf6d, user_id=7a58e4daf6024eb69fbae61e71e71e65, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:37.998222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.015555 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1287/2554] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:37 2022] PUT /identity/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d/users/7a58e4daf6024eb69fbae61e71e71e65/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 59 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:38.019347 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:38.030072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8bd9a08-b5e5-468f-9d5e-543dccbd5250 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:38.075374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:38.098272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:38.099068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.099354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.100465 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.101393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:38.103261 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20a4a9a7-26e2-4b4e-9cbe-cda35a60b717 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.113259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1268/2555] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:38 2022] GET /identity/v3/roles => generated 1420 bytes in 94 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:38.118374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:38.128933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0c6bf58f-c8f5-4da0-b64e-dac5058a8978 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:38.134004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:38.134911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.135209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.135474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] PATH_INFO: `/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d/users/7a58e4daf6024eb69fbae61e71e71e65/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.137415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=3c01bb6db1ec4ff3ab9423ccc9f5bf6d, user_id=7a58e4daf6024eb69fbae61e71e71e65, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:38.140122 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.150515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1288/2556] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:38 2022] PUT /identity/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d/users/7a58e4daf6024eb69fbae61e71e71e65/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:38.162594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.163009 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.163303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.190324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] MFA Rules not processed for user `7a58e4daf6024eb69fbae61e71e71e65`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:38.191583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:38.259575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1269/2557] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:38 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:38.270908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:38.273688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:38.330751 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa6ffbb9-31d0-4a01-b049-ec0904c9ccf7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:38.335146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:38.335786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.336070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.336345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.345480 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:38.346942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-941dedc6-0665-4ab7-9413-75bc538e0406 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.357154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1289/2558] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:38 2022] GET /identity/v3/auth/tokens => generated 3339 bytes in 87 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:38.503306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:38.566021 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb45a9fd-a317-4277-995a-409ec69887df None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:38.571541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:38.572417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.572701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.572980 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] PATH_INFO: `/v3/users/7a52eb7ff3bc46bb84b1ad364a718889` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.574159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a52eb7ff3bc46bb84b1ad364a718889)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:38.576132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.621401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a52eb7ff3bc46bb84b1ad364a718889'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:38.629018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a52eb7ff3bc46bb84b1ad364a718889'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:38.629724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a52eb7ff3bc46bb84b1ad364a718889'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:38.637854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a52eb7ff3bc46bb84b1ad364a718889'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:38.638985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:38.655290 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a52eb7ff3bc46bb84b1ad364a718889'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:38.661464 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1270/2559] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:38 2022] DELETE /identity/v3/users/7a52eb7ff3bc46bb84b1ad364a718889 => generated 0 bytes in 159 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:38.666152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.666651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.666752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.687318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:38.688713 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:38.739059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:38.749982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1290/2560] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:38 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:38.760350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:38.761822 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:38.819690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-db281d70-831b-45c8-8d66-662778789897 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:38.823409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:38.824054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:38.824339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:38.824621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:38.833690 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:38.835844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:38.845339 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1271/2561] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:38 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:39.109507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:39.111331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:39.129594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-82f0ce3b-879b-44c9-a97a-6724fd4215eb None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:39.135493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:39.136458 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.136805 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.137149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] PATH_INFO: `/v3/projects/a4e01a7e63cf46f08ecb2a90a71da826` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.138468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a4e01a7e63cf46f08ecb2a90a71da826)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:39.140871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:39.188550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a4e01a7e63cf46f08ecb2a90a71da826', 'request_id': 'req-8f43c565-b87e-42a6-b276-4a0619fe2726'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.196756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a4e01a7e63cf46f08ecb2a90a71da826', 'request_id': 'req-8f43c565-b87e-42a6-b276-4a0619fe2726'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.203376 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a4e01a7e63cf46f08ecb2a90a71da826', 'request_id': 'req-8f43c565-b87e-42a6-b276-4a0619fe2726'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.215319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] The token cache is being invalidate because project a4e01a7e63cf46f08ecb2a90a71da826 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:39.215609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.216268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.217896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1291/2562] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:39 2022] DELETE /identity/v3/projects/a4e01a7e63cf46f08ecb2a90a71da826 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:39.222192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:39.231878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dda87894-02df-4a0d-95c2-f2311a4fefb2 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:39.292557 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:39.293408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.293687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.293974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] PATH_INFO: `/v3/users/7a58e4daf6024eb69fbae61e71e71e65` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.295184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a58e4daf6024eb69fbae61e71e71e65)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:39.297106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:39.342331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a58e4daf6024eb69fbae61e71e71e65'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.350073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a58e4daf6024eb69fbae61e71e71e65'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.350731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a58e4daf6024eb69fbae61e71e71e65'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.357665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a58e4daf6024eb69fbae61e71e71e65'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.370061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-7be3a1d0-a600-43ba-9258-03311860a888 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a58e4daf6024eb69fbae61e71e71e65'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.375919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1272/2563] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:39 2022] DELETE /identity/v3/users/7a58e4daf6024eb69fbae61e71e71e65 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:39.650617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:39.652102 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:39.667594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f43c565-b87e-42a6-b276-4a0619fe2726 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:39.726841 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:39.727688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.727981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.728258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] PATH_INFO: `/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.729491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] RBAC: Authorizing `identity:delete_project(project_id=3c01bb6db1ec4ff3ab9423ccc9f5bf6d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:39.731535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:39.787896 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3c01bb6db1ec4ff3ab9423ccc9f5bf6d', 'request_id': 'req-ef81d194-2d9e-45cf-906d-950f796d9d5c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.796448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3c01bb6db1ec4ff3ab9423ccc9f5bf6d', 'request_id': 'req-ef81d194-2d9e-45cf-906d-950f796d9d5c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.803442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3c01bb6db1ec4ff3ab9423ccc9f5bf6d', 'request_id': 'req-ef81d194-2d9e-45cf-906d-950f796d9d5c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.815684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] The token cache is being invalidate because project 3c01bb6db1ec4ff3ab9423ccc9f5bf6d was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:39.815978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.816639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ef81d194-2d9e-45cf-906d-950f796d9d5c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:39.819299 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1292/2564] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:39 2022] DELETE /identity/v3/projects/3c01bb6db1ec4ff3ab9423ccc9f5bf6d => generated 0 bytes in 169 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:39.824561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.825054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.825248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.831736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:39.846618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:39.847923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:39.910565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1273/2565] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:39.919128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.919447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.919725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.939187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:39.940389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:39.949766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1293/2566] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:39 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:39.954118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:39.964242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5b0c35b4-e5d7-4839-8600-6f6dae7f644e None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:39.968862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:39.969518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:39.969809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:39.970084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:39.970832 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:39.971142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:39.972752 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:39.984952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1274/2567] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:39 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:07:39.989577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:39.996903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:40.002403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73b980de-656a-4e2f-a083-e2195a79a1ba None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.010714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.011766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.012416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.012759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.013574 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.015719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.046720 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1294/2568] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:39 2022] POST /identity/v3/projects => generated 373 bytes in 58 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:40.051108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.064175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae7fc26b-8890-4fef-aca5-3d141c822a14 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.129212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.129918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.130204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.130484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.131581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.133537 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.155427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1275/2569] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:40 2022] POST /identity/v3/users => generated 287 bytes in 105 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:40.159629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.170441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d4dd860-8897-4e36-860c-d12f7ab99d01 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.175175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.175884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.176164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.176576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.177254 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.179623 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.191267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1295/2570] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:40.196137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.206212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-579987df-a85c-4617-8c9f-bea57bc0afd9 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.211414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.212227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.212581 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.212953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.213764 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.215482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bb87de79-0af3-4612-93cd-4c47e38f5167 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.226364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1276/2571] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:40.230624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.243056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea393fec-ca05-4533-808f-7ef81d487351 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.247663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.248725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.249012 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.249292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] PATH_INFO: `/v3/projects/ad1202a4b89646d5949a981ec1435930/users/88bd892779ce4ec38be10276c86bf9f0/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.264331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] RBAC: Authorizing `identity:create_grant(project_id=ad1202a4b89646d5949a981ec1435930, user_id=88bd892779ce4ec38be10276c86bf9f0, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.267228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.299826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1296/2572] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:40 2022] PUT /identity/v3/projects/ad1202a4b89646d5949a981ec1435930/users/88bd892779ce4ec38be10276c86bf9f0/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 60 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:40.301729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.301929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.302207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.335355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] MFA Rules not processed for user `88bd892779ce4ec38be10276c86bf9f0`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:40.336570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:40.399707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:40.405508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1277/2573] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:40 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 105 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:40.410086 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.477422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8e4897ff-2570-4729-b593-4a466e69ec2e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.482151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.482844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.483135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.483414 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.484095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.486053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.510948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1297/2574] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:40 2022] POST /identity/v3/projects => generated 371 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:40.514882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.526371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-94e8b9c9-db84-4b27-a39a-f8941c0f9905 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.581235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.581902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.582193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.582476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.583204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.585638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.605444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1278/2575] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:40 2022] POST /identity/v3/users => generated 292 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:40.609469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.620449 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a69434b-bf5c-4679-bec6-4a5d9f50acfd admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.625233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.626040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.626306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.626581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.627342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.628982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.640423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1298/2576] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:40.644457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.655505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-57a3bf13-9f91-4e9a-bc95-6b8827d3db0b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.660197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.661084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.661374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.661657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] PATH_INFO: `/v3/projects/3cfbe95b928d4087890aa26d75ec87a5/users/5369d5aace8646568f0d96365429527f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.677160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cfbe95b928d4087890aa26d75ec87a5, user_id=5369d5aace8646568f0d96365429527f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.680288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.693089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1279/2577] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:40 2022] PUT /identity/v3/projects/3cfbe95b928d4087890aa26d75ec87a5/users/5369d5aace8646568f0d96365429527f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 49 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:40.697483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.707585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd4b0c4f-0b5e-4a8b-a556-be05a59b70a2 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.764067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.764648 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.764932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.765208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.765957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.767599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3871aaf1-fb06-4de0-a4ba-be88aeecd9c3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.777929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1299/2578] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:40 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:40.782311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.793394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b88994e9-8403-4a1c-a87d-b297869ae09f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:40.798421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:40.799335 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.799624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.799935 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] PATH_INFO: `/v3/projects/3cfbe95b928d4087890aa26d75ec87a5/users/5369d5aace8646568f0d96365429527f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.801924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] RBAC: Authorizing `identity:create_grant(project_id=3cfbe95b928d4087890aa26d75ec87a5, user_id=5369d5aace8646568f0d96365429527f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:40.804718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:40.814869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1280/2579] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:40 2022] PUT /identity/v3/projects/3cfbe95b928d4087890aa26d75ec87a5/users/5369d5aace8646568f0d96365429527f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:40.825586 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:40.825882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:40.826190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:40.853697 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] MFA Rules not processed for user `5369d5aace8646568f0d96365429527f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:40.854946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:40.860006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:40.924846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1300/2580] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:40 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:40.936514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:40.938248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:41.001799 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b102f7ab-493f-41f0-9d5e-f8e2a28b269e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:41.005437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:41.007159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:41.007444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:41.007726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:41.018131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:41.019632 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-abb53841-c346-4bee-8654-338b97d085fb service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:41.030132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1281/2581] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:40 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:41.103064 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:41.114726 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5dba46f8-288a-4dc8-82c7-37c99c3335bc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:41.119084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:41.119733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:41.120016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:41.120287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:41.180075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:41.181531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:41.190709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1301/2582] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:41 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:49.519452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:49.520004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:49.520126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:49.537806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] MFA Rules not processed for user `ac64444a4b794026a5e95584d31f1c04`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:49.539095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:49.585636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:49.594640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1282/2583] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:49 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 76 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:51.178971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:51.180588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:51.249506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-540b6676-2d9e-473f-a12b-0e0e14e95506 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:51.253711 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:51.254595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:51.255081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:51.255358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:51.260010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:51.323261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:51.335736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:51.337196 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:51.347485 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1302/2584] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:51 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 169 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:54.897378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:54.898954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:54.913508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1d03651d-9af3-4984-8b3e-264fe658a8cb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:54.918126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:54.919059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:54.919348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:54.919629 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] PATH_INFO: `/v3/users/9a9256b6d7e24c54b061673cd17217cd` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:54.920820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] RBAC: Authorizing `identity:delete_user(user_id=9a9256b6d7e24c54b061673cd17217cd)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:54.922762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:54.973580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '9a9256b6d7e24c54b061673cd17217cd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:54.981452 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '9a9256b6d7e24c54b061673cd17217cd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:54.982204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '9a9256b6d7e24c54b061673cd17217cd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:54.989626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '9a9256b6d7e24c54b061673cd17217cd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.003330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-35494774-d1a7-4c14-a399-356a32a91594 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '9a9256b6d7e24c54b061673cd17217cd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.009967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1283/2585] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:54 2022] DELETE /identity/v3/users/9a9256b6d7e24c54b061673cd17217cd => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:55.223438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.233892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55fda301-2652-4366-8ef1-4146248c225d service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.294930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.295792 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.296081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.296357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] PATH_INFO: `/v3/projects/75293bf0100f435bb567b690edc85533` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.297552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] RBAC: Authorizing `identity:delete_project(project_id=75293bf0100f435bb567b690edc85533)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.299522 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.349025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '75293bf0100f435bb567b690edc85533', 'request_id': 'req-5efb7fc8-a6b5-486b-842a-89c8167d15ec'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.356710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '75293bf0100f435bb567b690edc85533', 'request_id': 'req-5efb7fc8-a6b5-486b-842a-89c8167d15ec'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.363203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '75293bf0100f435bb567b690edc85533', 'request_id': 'req-5efb7fc8-a6b5-486b-842a-89c8167d15ec'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.374948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] The token cache is being invalidate because project 75293bf0100f435bb567b690edc85533 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:55.375243 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.375921 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-5efb7fc8-a6b5-486b-842a-89c8167d15ec admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:55.377474 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1303/2586] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:55 2022] DELETE /identity/v3/projects/75293bf0100f435bb567b690edc85533 => generated 0 bytes in 155 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:55.383356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.383648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.383951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.402974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:55.404211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:55.468846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1284/2587] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:07:55.476543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.476854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.477131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.495415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:55.496599 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:55.504379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1304/2588] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:55.509201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.518920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4d2df58d-7f96-409f-b9e6-350e1b7638dd None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.523582 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.524224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.524503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.524776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.525524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:07:55.525827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.527449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.538851 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1285/2589] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:07:55 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:07:55.543204 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.552443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e21ed13c-5b79-43c1-a5a2-8be18e54179c None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.557812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.558455 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.558739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.559049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.559753 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.561666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.589764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1305/2590] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:55 2022] POST /identity/v3/projects => generated 405 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:55.593315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.603158 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd7f4669-45e7-4b9b-bd0f-b90f7661091e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.653187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:55.660317 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.661007 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.661599 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.662040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.663083 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.665666 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.690565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1286/2591] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:55 2022] POST /identity/v3/users => generated 303 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:55.691758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.702947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43b98020-e339-4bd7-b573-fe8b7feafe09 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.707887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.708579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.708866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.709129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.709852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.711524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.722105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1306/2592] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:55.728670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.738967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a588804-1003-4c44-875c-e3431f5c0c3a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.744312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.744989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.745263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.745569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.746293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.747931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7090a1-3446-48e7-b0cd-0073b4aa2367 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.758947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1287/2593] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:55 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:55.763126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.773234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c477aee6-2535-4b4d-aa3b-32a1e0e799e0 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.778435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.779478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.779827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.780177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] PATH_INFO: `/v3/projects/61cc3e13d49a455aa7253d1ab74d392f/users/61694d31ae534d39a32f805a20976675/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.796190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] RBAC: Authorizing `identity:create_grant(project_id=61cc3e13d49a455aa7253d1ab74d392f, user_id=61694d31ae534d39a32f805a20976675, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:55.798980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:55.812955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1307/2594] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:55 2022] PUT /identity/v3/projects/61cc3e13d49a455aa7253d1ab74d392f/users/61694d31ae534d39a32f805a20976675/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:55.818272 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7613b-389a-4aa5-9b3f-3279edecd0c2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.818827 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7613b-389a-4aa5-9b3f-3279edecd0c2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.819150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3fb7613b-389a-4aa5-9b3f-3279edecd0c2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:55.837271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3fb7613b-389a-4aa5-9b3f-3279edecd0c2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:55.838624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3fb7613b-389a-4aa5-9b3f-3279edecd0c2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:55.904213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1288/2595] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:55 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:55.914470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:55.916316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:55.941953 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:55.994158 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae42347b-64f0-4985-8853-0f231ea53c37 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:55.998116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:55.998892 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:55.999270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:55.999671 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:56.010128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:56.011695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:56.022384 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1308/2596] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:07:55 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:07:56.741556 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:56.741917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:56.742411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:56.768807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] MFA Rules not processed for user `61694d31ae534d39a32f805a20976675`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:56.770026 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:56.839369 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1289/2597] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:56 2022] POST /identity/v3/auth/tokens => generated 3272 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:56.843967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:56.854510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-389886cf-9079-4921-a9f3-67ba15ea582d service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:56.859368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:56.859960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:56.860242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:56.860519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:56.861268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:56.863308 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:56.895572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1309/2598] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:07:56 2022] POST /identity/v3/projects => generated 405 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:07:56.897669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:56.908712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bae1bbbc-4654-4f1b-a8a9-5fd24d36a4b3 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:56.969035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:56.969716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:56.970003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:56.970313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:56.971081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:56.973040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:56.983304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:56.997562 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1290/2599] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:07:56 2022] POST /identity/v3/users => generated 309 bytes in 98 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:07:56.998825 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:57.009901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8d7a7461-055c-47b8-9a23-123efa7064ad admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:57.014908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:57.015652 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:57.016003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:57.016349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:57.017083 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:57.018694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:57.032537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1310/2600] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:56 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:07:57.034294 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:57.049156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fab7d690-3ddb-4f28-920c-b6876d917ef7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:57.054669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:57.055600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:57.055893 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:57.056171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] PATH_INFO: `/v3/projects/6c2e2387023f407ba83bb828f85da256/users/678ca80ded8a4db9a3f20d599b3ef183/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:57.074309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c2e2387023f407ba83bb828f85da256, user_id=678ca80ded8a4db9a3f20d599b3ef183, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:57.077246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:57.096270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1291/2601] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:57 2022] PUT /identity/v3/projects/6c2e2387023f407ba83bb828f85da256/users/678ca80ded8a4db9a3f20d599b3ef183/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 63 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:57.100421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:57.113911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17dc2057-2fe1-4dd2-b0f3-b3340b7a2b10 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:57.194760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:57.195935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:57.196434 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:57.196943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:57.198916 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:57.201984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69002474-1123-49d2-bf97-9b696ea3ffad admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:57.223468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1311/2602] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:07:57 2022] GET /identity/v3/roles => generated 1420 bytes in 124 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:07:57.224433 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:57.245370 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69ea289e-5d39-4baa-9908-2bc25106df1a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:57.253308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:57.254982 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:57.255652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:57.256279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] PATH_INFO: `/v3/projects/6c2e2387023f407ba83bb828f85da256/users/678ca80ded8a4db9a3f20d599b3ef183/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:57.263442 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6c2e2387023f407ba83bb828f85da256, user_id=678ca80ded8a4db9a3f20d599b3ef183, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:57.267123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:57.284620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1292/2603] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:07:57 2022] PUT /identity/v3/projects/6c2e2387023f407ba83bb828f85da256/users/678ca80ded8a4db9a3f20d599b3ef183/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:07:57.964898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:57.965418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:57.965515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:57.992116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] MFA Rules not processed for user `678ca80ded8a4db9a3f20d599b3ef183`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:07:57.993345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:58.070749 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1312/2604] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:07:57 2022] POST /identity/v3/auth/tokens => generated 3339 bytes in 107 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:07:59.236522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:59.314911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-06f147cc-25bd-4dd8-8fb1-e9bdda878a82 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:59.320224 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:59.321155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:59.321471 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:59.321902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] PATH_INFO: `/v3/users/61694d31ae534d39a32f805a20976675` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:59.323197 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] RBAC: Authorizing `identity:delete_user(user_id=61694d31ae534d39a32f805a20976675)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:59.325187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:59.378396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '61694d31ae534d39a32f805a20976675'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.388638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '61694d31ae534d39a32f805a20976675'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.389328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '61694d31ae534d39a32f805a20976675'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.398300 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '61694d31ae534d39a32f805a20976675'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.420968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '61694d31ae534d39a32f805a20976675'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.431577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1293/2605] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:59 2022] DELETE /identity/v3/users/61694d31ae534d39a32f805a20976675 => generated 0 bytes in 193 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:59.618344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:59.629031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1cb0308e-8948-4e10-994a-2b95b9a116f3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:59.635050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:59.689613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:59.690483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:59.690790 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:59.691071 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] PATH_INFO: `/v3/projects/61cc3e13d49a455aa7253d1ab74d392f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:59.692265 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] RBAC: Authorizing `identity:delete_project(project_id=61cc3e13d49a455aa7253d1ab74d392f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:59.694698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:59.739337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '61cc3e13d49a455aa7253d1ab74d392f', 'request_id': 'req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.746613 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '61cc3e13d49a455aa7253d1ab74d392f', 'request_id': 'req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.752415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '61cc3e13d49a455aa7253d1ab74d392f', 'request_id': 'req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.765007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] The token cache is being invalidate because project 61cc3e13d49a455aa7253d1ab74d392f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:07:59.765392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.766229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.768082 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1313/2606] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:07:59 2022] DELETE /identity/v3/projects/61cc3e13d49a455aa7253d1ab74d392f => generated 0 bytes in 150 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:07:59.772515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:07:59.773919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:07:59.786866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4de83abd-52c8-4194-99e7-f13e7a224b2b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:07:59.821031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=87,keystone:INSERT=3,keystone:DELETE=10 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:07:59.844546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:07:59.845412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:07:59.845695 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:07:59.845968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] PATH_INFO: `/v3/users/678ca80ded8a4db9a3f20d599b3ef183` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:07:59.847182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] RBAC: Authorizing `identity:delete_user(user_id=678ca80ded8a4db9a3f20d599b3ef183)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:07:59.849127 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:07:59.899106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '678ca80ded8a4db9a3f20d599b3ef183'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.908023 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '678ca80ded8a4db9a3f20d599b3ef183'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.908634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '678ca80ded8a4db9a3f20d599b3ef183'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.916602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '678ca80ded8a4db9a3f20d599b3ef183'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.929072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-1643d399-4921-4151-a030-c6bb2e08bb4a admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '678ca80ded8a4db9a3f20d599b3ef183'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:07:59.935277 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1294/2607] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:07:59 2022] DELETE /identity/v3/users/678ca80ded8a4db9a3f20d599b3ef183 => generated 0 bytes in 163 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:00.153659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.155689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:00.169483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1980bb2a-e1c8-46ce-8b91-d84d6aa0adcc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.225903 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.226799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.227093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.227370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] PATH_INFO: `/v3/projects/6c2e2387023f407ba83bb828f85da256` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.228590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6c2e2387023f407ba83bb828f85da256)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.230545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.270365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:00.279068 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6c2e2387023f407ba83bb828f85da256', 'request_id': 'req-38cedb45-909a-46fd-bf46-8507dd884970'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:00.286573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6c2e2387023f407ba83bb828f85da256', 'request_id': 'req-38cedb45-909a-46fd-bf46-8507dd884970'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:00.293133 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6c2e2387023f407ba83bb828f85da256', 'request_id': 'req-38cedb45-909a-46fd-bf46-8507dd884970'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:00.304084 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] The token cache is being invalidate because project 6c2e2387023f407ba83bb828f85da256 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:00.304370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:00.304995 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-38cedb45-909a-46fd-bf46-8507dd884970 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:00.306634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1314/2608] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:00 2022] DELETE /identity/v3/projects/6c2e2387023f407ba83bb828f85da256 => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:00.311619 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.312119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.312226 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.330513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:00.331651 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:00.390746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1295/2609] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:00.399175 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.399464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.399741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.418054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:00.419287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:00.427968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1315/2610] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:00.432409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.441979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d1e6b12-b239-47ae-928f-3a1c8d6bbf28 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.447497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.448147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.448432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.448711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.449434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:00.449747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.451390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.471330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1296/2611] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:00 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:00.473009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.485595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c0e0cf82-1b9a-43b8-9ccf-61ce069fc61b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.490303 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.491003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.491289 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.491571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.492292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.494220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.520727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1316/2612] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/projects => generated 369 bytes in 49 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:00.525686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.535932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bf432702-573a-43d1-97c9-fe01d9af486f admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.582132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:00.600945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.601781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.601888 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.602160 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.602891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.604862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.628054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1297/2613] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/users => generated 285 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:00.629193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.641114 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b772b1ad-24ce-45fe-97a6-2a2773edf020 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.645899 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.647401 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.647704 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.647986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.648756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.650479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.667189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1317/2614] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:00.668005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.678946 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7d21ae2-cff0-4c83-a926-6528033afa88 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.683744 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.684315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.684624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.684908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.685640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.687285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-40b66986-8122-4b46-adc0-a522c1ce7b65 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.699142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1298/2615] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:00 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:00.703223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.713222 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20da4864-e4d6-47ca-a048-a4f37e7d7fa3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.717849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.718718 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.719030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.719301 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] PATH_INFO: `/v3/projects/036ff20989b04219b848e9e9b2ba524f/users/7a76a7bad41547e196b24406809eea22/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.734094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] RBAC: Authorizing `identity:create_grant(project_id=036ff20989b04219b848e9e9b2ba524f, user_id=7a76a7bad41547e196b24406809eea22, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.736776 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.749636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1318/2616] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:00 2022] PUT /identity/v3/projects/036ff20989b04219b848e9e9b2ba524f/users/7a76a7bad41547e196b24406809eea22/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:00.760701 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.760998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.761286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.788218 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] MFA Rules not processed for user `7a76a7bad41547e196b24406809eea22`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:00.789409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:00.853652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1299/2617] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/auth/tokens => generated 3236 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:00.857739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.925242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c04e21a7-b80b-46dd-bb0f-bc91aad03e4c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:00.930552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:00.931283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:00.931569 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:00.931849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:00.932627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:00.934561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:00.961482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1319/2618] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/projects => generated 369 bytes in 104 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:00.965564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:00.978618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-acafc0bc-39bf-438c-8e19-9ff2eb3a159b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.035151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.035816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.036099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.036375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.037650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.039638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.064640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1300/2619] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:00 2022] POST /identity/v3/users => generated 291 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:01.066278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:01.077101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2936bf57-00e5-4157-9c70-aa9b29bb9984 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.081719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.082389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.082694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.082982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.083708 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.085585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.099169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1320/2620] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:01 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:01.101491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:01.111657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-512a57b6-04a4-4408-89c2-9c8b923c98c0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.117050 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.117948 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.118237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.118551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] PATH_INFO: `/v3/projects/8a92592d636440c7940d3ed7a982454f/users/85f3b0b3cdcd4b238eb279b830300fba/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.134056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a92592d636440c7940d3ed7a982454f, user_id=85f3b0b3cdcd4b238eb279b830300fba, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.136817 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.150069 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1301/2621] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:01 2022] PUT /identity/v3/projects/8a92592d636440c7940d3ed7a982454f/users/85f3b0b3cdcd4b238eb279b830300fba/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:01.154162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:01.164972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-25612089-c6fc-4fac-8e55-d2cf572bb715 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.203248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=4,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:01.223030 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.223710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.223994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.224272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.225001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.227180 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-af62156d-67c0-4d7e-b203-b0a2432a1f0d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.237251 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1321/2622] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:01 2022] GET /identity/v3/roles => generated 1420 bytes in 84 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:01.241968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:01.252643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-00e6e571-894a-44fc-b5d2-9608df4567b8 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.258114 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.259067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.259334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.259611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] PATH_INFO: `/v3/projects/8a92592d636440c7940d3ed7a982454f/users/85f3b0b3cdcd4b238eb279b830300fba/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.261575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] RBAC: Authorizing `identity:create_grant(project_id=8a92592d636440c7940d3ed7a982454f, user_id=85f3b0b3cdcd4b238eb279b830300fba, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.264242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.278640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1302/2623] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:01 2022] PUT /identity/v3/projects/8a92592d636440c7940d3ed7a982454f/users/85f3b0b3cdcd4b238eb279b830300fba/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 38 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:01.290330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.290488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.290794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.320290 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] MFA Rules not processed for user `85f3b0b3cdcd4b238eb279b830300fba`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:01.321514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:01.391829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1322/2624] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:01 2022] POST /identity/v3/auth/tokens => generated 3303 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:01.403539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:01.405428 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:01.412207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:01.469229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb7c7eef-9d0a-49ab-bef7-1c21944d5b86 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:01.473124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:01.473769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:01.474058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:01.474341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:01.484422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:01.486054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c131dfdf-3f3b-4af1-a32d-aa90100b6f8c service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:01.496088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1303/2625] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:01 2022] GET /identity/v3/auth/tokens => generated 3303 bytes in 93 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:02.915337 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:02.977696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a1abc7fc-7643-4f66-9bc9-483eb80f80e4 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:02.982073 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:02.982970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:02.983246 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:02.983515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] PATH_INFO: `/v3/users/7a76a7bad41547e196b24406809eea22` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:02.984687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7a76a7bad41547e196b24406809eea22)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:02.986598 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:03.031549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7a76a7bad41547e196b24406809eea22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.039479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7a76a7bad41547e196b24406809eea22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.040093 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7a76a7bad41547e196b24406809eea22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.047008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7a76a7bad41547e196b24406809eea22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.047640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7a76a7bad41547e196b24406809eea22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.049505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=91,keystone:DELETE=8,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:03.066663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1323/2626] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:02 2022] DELETE /identity/v3/users/7a76a7bad41547e196b24406809eea22 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:03.071555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:03.072028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:03.072146 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:03.089346 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:03.090539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:03.151148 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1304/2627] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:03 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:03.161850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:03.164152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:03.224562 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ab3e4c5d-da6b-43da-a77b-4ab8ea842216 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:03.228666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:03.229253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:03.229548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:03.229819 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:03.238942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:03.240383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:03.251223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1324/2628] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:03 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:03.528986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:03.530491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:03.545800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ccde8684-af0b-481f-8bda-928dcf26a61d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:03.550328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:03.551225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:03.551506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:03.551781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] PATH_INFO: `/v3/projects/036ff20989b04219b848e9e9b2ba524f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:03.554920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] RBAC: Authorizing `identity:delete_project(project_id=036ff20989b04219b848e9e9b2ba524f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:03.557059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:03.602236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '036ff20989b04219b848e9e9b2ba524f', 'request_id': 'req-b14a6cd4-b2ad-46b3-ab0c-295292003f55'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.607641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '036ff20989b04219b848e9e9b2ba524f', 'request_id': 'req-b14a6cd4-b2ad-46b3-ab0c-295292003f55'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.614865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '036ff20989b04219b848e9e9b2ba524f', 'request_id': 'req-b14a6cd4-b2ad-46b3-ab0c-295292003f55'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.626413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] The token cache is being invalidate because project 036ff20989b04219b848e9e9b2ba524f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:03.626703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.627360 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.628899 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1305/2629] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:03 2022] DELETE /identity/v3/projects/036ff20989b04219b848e9e9b2ba524f => generated 0 bytes in 100 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:03.633033 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:03.642897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a17b6d5-6ca9-41e3-91cc-c15b819a93c7 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:03.699468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:03.700331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:03.700608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:03.700880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] PATH_INFO: `/v3/users/85f3b0b3cdcd4b238eb279b830300fba` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:03.702045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] RBAC: Authorizing `identity:delete_user(user_id=85f3b0b3cdcd4b238eb279b830300fba)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:03.703990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:03.752438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '85f3b0b3cdcd4b238eb279b830300fba'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.759797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '85f3b0b3cdcd4b238eb279b830300fba'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.760406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '85f3b0b3cdcd4b238eb279b830300fba'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.767329 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '85f3b0b3cdcd4b238eb279b830300fba'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.767958 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-b8ee95b8-3466-4fd2-9749-0417d15d8811 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '85f3b0b3cdcd4b238eb279b830300fba'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:03.784928 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1325/2630] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:03 2022] DELETE /identity/v3/users/85f3b0b3cdcd4b238eb279b830300fba => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:04.050696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.052365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:04.066315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b14a6cd4-b2ad-46b3-ab0c-295292003f55 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.095494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:04.125389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.126334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.126623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.126937 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] PATH_INFO: `/v3/projects/8a92592d636440c7940d3ed7a982454f` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.128118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] RBAC: Authorizing `identity:delete_project(project_id=8a92592d636440c7940d3ed7a982454f)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.130093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.177794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '8a92592d636440c7940d3ed7a982454f', 'request_id': 'req-a0106d87-071a-412a-bc27-db4530932320'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:04.184282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '8a92592d636440c7940d3ed7a982454f', 'request_id': 'req-a0106d87-071a-412a-bc27-db4530932320'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:04.192230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '8a92592d636440c7940d3ed7a982454f', 'request_id': 'req-a0106d87-071a-412a-bc27-db4530932320'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:04.203685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] The token cache is being invalidate because project 8a92592d636440c7940d3ed7a982454f was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:04.203975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:04.204600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a0106d87-071a-412a-bc27-db4530932320 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:04.206147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1306/2631] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:04 2022] DELETE /identity/v3/projects/8a92592d636440c7940d3ed7a982454f => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:04.211059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.211533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.211631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.217877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=87,keystone:DELETE=12,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:04.232570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:04.233769 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:04.295200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1326/2632] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:04.302686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.303087 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.303485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.323227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:04.324448 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:04.332456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1307/2633] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 31 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:04.337323 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.347089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ae85dd69-e780-48f6-b6c8-78fa042150ae None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.351875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.352505 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.352786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.353057 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.354278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:04.354582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.356230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.368779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1327/2634] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:04 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:04.373188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.383222 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7c0e6285-b185-444e-b288-28d3318da8a1 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.387880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.388535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.388820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.389097 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.389795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.391748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.424070 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1308/2635] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/projects => generated 381 bytes in 51 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:04.428238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.442476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c46b1620-0729-429a-8d85-54830465556e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.502431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.503129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.503412 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.503684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.504370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.506291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.526907 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1328/2636] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/users => generated 291 bytes in 99 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:04.531208 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.541432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7981861a-7236-41bb-a114-eca70d4b17f3 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.546167 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.546788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.547089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.547372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.548115 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.549715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.560516 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1309/2637] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:04.564363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.575238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-825abfa0-3e57-464b-b1ef-4227095dc8a9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.579927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.580579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.580864 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.581150 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.581869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.583513 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b141ea0d-5001-49ff-83ee-9df16affebe9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.594056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1329/2638] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:04 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:04.598056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.608568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c4a7fbfb-f60a-4a53-aa4b-7918c6042aab admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.613203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.614060 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.614344 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.614621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] PATH_INFO: `/v3/projects/c5a8e3d224b447669730b1364b523a4c/users/04f6b0fa90fe46f7ac9c563b5dea0e09/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.629390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] RBAC: Authorizing `identity:create_grant(project_id=c5a8e3d224b447669730b1364b523a4c, user_id=04f6b0fa90fe46f7ac9c563b5dea0e09, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.632682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.645258 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1310/2639] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:04 2022] PUT /identity/v3/projects/c5a8e3d224b447669730b1364b523a4c/users/04f6b0fa90fe46f7ac9c563b5dea0e09/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:04.656444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.656733 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.657009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.684353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] MFA Rules not processed for user `04f6b0fa90fe46f7ac9c563b5dea0e09`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:04.685655 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:04.744875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:04.751520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1330/2640] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/auth/tokens => generated 3248 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:04.756245 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.815108 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:04.819675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ece21e48-e99e-44ff-a5a8-3759de8743fa admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.826043 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.826831 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.827328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.827624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.828541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.830714 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.855523 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1311/2641] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/projects => generated 381 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:04.859559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.871286 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1a2f0606-5d8a-46de-9d6c-bdb0ce831c4a None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.926238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.926940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.927229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.927507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.928201 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.930802 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.951085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1331/2642] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:04 2022] POST /identity/v3/users => generated 297 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:04.955164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:04.966005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-217c95a4-aab1-449e-a2ca-51eb17d62ba6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:04.970667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:04.971353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:04.971640 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:04.971917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:04.972637 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:04.974214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:04.985398 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1312/2643] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:04 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:04.989436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.000079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0a20be69-0015-4018-a184-978468f8b23c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.004740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.005600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.005879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.006151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] PATH_INFO: `/v3/projects/ae65df3e8e204b16bc5b60c8997842ac/users/e8c2508968e246738bc44708a66eeb41/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.021035 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae65df3e8e204b16bc5b60c8997842ac, user_id=e8c2508968e246738bc44708a66eeb41, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.024025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.037100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1332/2644] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:04 2022] PUT /identity/v3/projects/ae65df3e8e204b16bc5b60c8997842ac/users/e8c2508968e246738bc44708a66eeb41/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:05.041794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.051691 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d96cd96f-c1ce-49fa-acde-853d2f5e351c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.108836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.109478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.109694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.109974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.110689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.112377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6b3372a6-9b0d-4f20-a292-5edf5d677eaf admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.122517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1313/2645] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:05 2022] GET /identity/v3/roles => generated 1420 bytes in 81 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:05.126622 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.136876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d1962f79-5426-45d6-a793-20ca25dbfebc admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.141493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.142395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.142676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.142980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] PATH_INFO: `/v3/projects/ae65df3e8e204b16bc5b60c8997842ac/users/e8c2508968e246738bc44708a66eeb41/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.144973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] RBAC: Authorizing `identity:create_grant(project_id=ae65df3e8e204b16bc5b60c8997842ac, user_id=e8c2508968e246738bc44708a66eeb41, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.148192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.158389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1333/2646] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:05 2022] PUT /identity/v3/projects/ae65df3e8e204b16bc5b60c8997842ac/users/e8c2508968e246738bc44708a66eeb41/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:05.169336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.169634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.169914 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.196262 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] MFA Rules not processed for user `e8c2508968e246738bc44708a66eeb41`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:05.197742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:05.265668 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1314/2647] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:05 2022] POST /identity/v3/auth/tokens => generated 3315 bytes in 97 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:05.278541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.280339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:05.337391 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b84173d5-166e-4eae-a49d-4ef0f5f5d56d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.341287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.341935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.342223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.342501 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.351471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.352906 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.362295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1334/2648] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:05 2022] GET /identity/v3/auth/tokens => generated 3315 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:05.675639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.685444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa0254da-3bac-4179-b100-21b93fdd8446 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.689957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8c2508968e246738bc44708a66eeb41', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae65df3e8e204b16bc5b60c8997842ac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.690614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.690947 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.691229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.691954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:05.692265 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.694124 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.705548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1315/2649] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:05.709736 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.719074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-90383c07-fb02-46a2-bfad-684717bac903 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.723688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8c2508968e246738bc44708a66eeb41', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae65df3e8e204b16bc5b60c8997842ac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.724368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.724660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.724942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.726146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.728230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.754515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1335/2650] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:05 2022] POST /identity/v3/projects => generated 354 bytes in 45 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:05.758794 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.768748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14c63f95-3ccb-4a89-aa06-5449cd725db7 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.806574 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:05.826404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8c2508968e246738bc44708a66eeb41', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae65df3e8e204b16bc5b60c8997842ac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.827099 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.827412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.827694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.828406 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:05.828721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.830302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.841627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1316/2651] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:05 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 83 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:08:05.881007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:05.891219 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e2abf28f-fb1c-46f8-af00-d68b39e86c96 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:05.895827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8c2508968e246738bc44708a66eeb41', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ae65df3e8e204b16bc5b60c8997842ac', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:05.896683 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:05.896975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:05.897258 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] PATH_INFO: `/v3/projects/a15d5f5b55bb4c189fe78f29d7279ced` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:05.899034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorizing `identity:delete_project(project_id=a15d5f5b55bb4c189fe78f29d7279ced)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:05.900939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:05.931464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=93,keystone:INSERT=6,keystone:DELETE=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:05.950639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a15d5f5b55bb4c189fe78f29d7279ced', 'request_id': 'req-79f7125c-37fd-4973-8120-bd6a7d571a47'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:05.957820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a15d5f5b55bb4c189fe78f29d7279ced', 'request_id': 'req-79f7125c-37fd-4973-8120-bd6a7d571a47'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:05.964214 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a15d5f5b55bb4c189fe78f29d7279ced', 'request_id': 'req-79f7125c-37fd-4973-8120-bd6a7d571a47'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:05.975006 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] The token cache is being invalidate because project a15d5f5b55bb4c189fe78f29d7279ced was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:05.975296 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:05.976591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-79f7125c-37fd-4973-8120-bd6a7d571a47 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:05.978191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1336/2652] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:05 2022] DELETE /identity/v3/projects/a15d5f5b55bb4c189fe78f29d7279ced => generated 0 bytes in 98 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:06.168202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:06.169945 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:06.233833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53509f42-ec57-4387-9a34-7483cf596631 tempest-VolumeQuotaClassesTest-1801710611 tempest-VolumeQuotaClassesTest-1801710611-project-admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:06.291991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:06.292875 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:06.293155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:06.293430 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] PATH_INFO: `/v3/users/04f6b0fa90fe46f7ac9c563b5dea0e09` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:06.294611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] RBAC: Authorizing `identity:delete_user(user_id=04f6b0fa90fe46f7ac9c563b5dea0e09)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:06.296572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:06.341918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '04f6b0fa90fe46f7ac9c563b5dea0e09'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.349012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '04f6b0fa90fe46f7ac9c563b5dea0e09'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.349639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '04f6b0fa90fe46f7ac9c563b5dea0e09'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.357426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '04f6b0fa90fe46f7ac9c563b5dea0e09'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.369559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '04f6b0fa90fe46f7ac9c563b5dea0e09'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.371639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:06.376293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1317/2653] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:06 2022] DELETE /identity/v3/users/04f6b0fa90fe46f7ac9c563b5dea0e09 => generated 0 bytes in 209 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:06.380847 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:06.381313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:06.381427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:06.399181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:06.400320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:06.459635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1337/2654] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:06 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 80 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:06.469313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:06.470718 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:06.527595 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9fe23234-c22d-4a56-9c7a-eba72ec1412d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:06.531248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:06.531885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:06.532171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:06.532450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:06.541915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:06.543366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:06.552659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1318/2655] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:06.821483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:06.822985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:06.837939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-987c92f7-97fb-46f1-bcc6-c80a5027ab15 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:06.843203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:06.844161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:06.844506 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:06.844849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] PATH_INFO: `/v3/projects/c5a8e3d224b447669730b1364b523a4c` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:06.847354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c5a8e3d224b447669730b1364b523a4c)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:06.849508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:06.897344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c5a8e3d224b447669730b1364b523a4c', 'request_id': 'req-6cb03be6-07ec-410a-b755-5706d8eee89c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.905230 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c5a8e3d224b447669730b1364b523a4c', 'request_id': 'req-6cb03be6-07ec-410a-b755-5706d8eee89c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.910610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c5a8e3d224b447669730b1364b523a4c', 'request_id': 'req-6cb03be6-07ec-410a-b755-5706d8eee89c'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.922518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] The token cache is being invalidate because project c5a8e3d224b447669730b1364b523a4c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:06.922845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.923504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:06.925105 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1338/2656] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:06 2022] DELETE /identity/v3/projects/c5a8e3d224b447669730b1364b523a4c => generated 0 bytes in 104 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:06.929315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:06.939052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ed83eb5f-033a-48bd-b027-a45ed996733c service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:06.995705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:06.996580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:06.996867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:06.997150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] PATH_INFO: `/v3/users/e8c2508968e246738bc44708a66eeb41` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:06.998343 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] RBAC: Authorizing `identity:delete_user(user_id=e8c2508968e246738bc44708a66eeb41)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.000298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.050038 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'e8c2508968e246738bc44708a66eeb41'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.058862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'e8c2508968e246738bc44708a66eeb41'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.059489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'e8c2508968e246738bc44708a66eeb41'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.066371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'e8c2508968e246738bc44708a66eeb41'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.078387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5576ba24-ab73-42e0-8d05-f51548996499 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'e8c2508968e246738bc44708a66eeb41'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.084324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1319/2657] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:06 2022] DELETE /identity/v3/users/e8c2508968e246738bc44708a66eeb41 => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:07.355374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.356942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:07.370986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cb03be6-07ec-410a-b755-5706d8eee89c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.404699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=13,keystone:SELECT=85,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:07.430338 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.431260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.431548 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.431828 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] PATH_INFO: `/v3/projects/ae65df3e8e204b16bc5b60c8997842ac` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.433046 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] RBAC: Authorizing `identity:delete_project(project_id=ae65df3e8e204b16bc5b60c8997842ac)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.435014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.481688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ae65df3e8e204b16bc5b60c8997842ac', 'request_id': 'req-a4079ea9-4027-4497-afd7-9b81d94ee7fb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.488633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ae65df3e8e204b16bc5b60c8997842ac', 'request_id': 'req-a4079ea9-4027-4497-afd7-9b81d94ee7fb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.494052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ae65df3e8e204b16bc5b60c8997842ac', 'request_id': 'req-a4079ea9-4027-4497-afd7-9b81d94ee7fb'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.504695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] The token cache is being invalidate because project ae65df3e8e204b16bc5b60c8997842ac was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:07.505015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.505675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a4079ea9-4027-4497-afd7-9b81d94ee7fb admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:07.507264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1339/2658] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:07 2022] DELETE /identity/v3/projects/ae65df3e8e204b16bc5b60c8997842ac => generated 0 bytes in 153 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:07.513030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.513504 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.513626 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.531425 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:07.532635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:07.549859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:07.594109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1320/2659] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:07.601963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.602252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.602530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.620999 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:07.622208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:07.630021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1340/2660] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:07.634790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.644429 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e6c17a56-05bc-4816-a517-793006cddabb None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.649236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.649886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.650169 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.650456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.651220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:07.651533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.653130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.664721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1321/2661] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:07.669112 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.678241 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b076573-539a-454f-a846-950cfdcf5008 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.683288 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.683926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.684203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.684472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.685200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.687125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.712875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1341/2662] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:07 2022] POST /identity/v3/projects => generated 377 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:07.717166 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.727532 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4c91afb2-0261-4a8d-85a0-3480de2dad05 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.785242 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.785922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.786539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.786539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.787211 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.789772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.811109 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1322/2663] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:07 2022] POST /identity/v3/users => generated 289 bytes in 94 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:07.815139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.827312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c09d0e78-135a-42d4-b069-cf0df68f324f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.832345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.833124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.833577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.833850 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.834715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.836662 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.848454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1342/2664] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:07.852917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.863928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-980fe946-e714-4c96-bd47-31107af9f29d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.870843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.871625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.871981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.873028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.873175 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.874863 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-29108c52-7266-4e3d-aaf5-aa88e9848225 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.888774 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1323/2665] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:07 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:07.892854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:07.904862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa34b9fd-495d-4679-ad97-e24c99e3164a admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:07.910973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:07.913601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.913601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.913601 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] PATH_INFO: `/v3/projects/d44bf1b03a7d4d4190165b383dec5a1c/users/c7e7c6043fad414d89db086df799311b/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:07.933578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] RBAC: Authorizing `identity:create_grant(project_id=d44bf1b03a7d4d4190165b383dec5a1c, user_id=c7e7c6043fad414d89db086df799311b, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:07.936719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:07.954863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1343/2666] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:07 2022] PUT /identity/v3/projects/d44bf1b03a7d4d4190165b383dec5a1c/users/c7e7c6043fad414d89db086df799311b/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:07.967412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:07.967636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:07.967991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.004671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] MFA Rules not processed for user `c7e7c6043fad414d89db086df799311b`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:08.005981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:08.082842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1324/2667] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:07 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 115 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:08.086091 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.153209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a92edb15-2398-461a-a8cd-8310e4ac0b9c admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.158032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.159231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.159231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.159517 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.160194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.162169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.176649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:08.196264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1344/2668] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:08 2022] POST /identity/v3/projects => generated 375 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:08.200594 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.213481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-87595faa-f7c6-46a3-be2a-7bf3dcfdcaf6 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.228881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:08.280235 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.280927 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.281215 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.281493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.282179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.284153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.313748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1325/2669] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:08 2022] POST /identity/v3/users => generated 288 bytes in 111 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:08.314869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.326577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26bcf551-608a-42bb-8712-db3a43e22fc7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.333067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.333744 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.334054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.334373 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.335415 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.337682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.350147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1345/2670] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:08 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:08.354291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.367614 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-52412d8b-4d21-4771-bb3c-58e504b6e0a6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.373680 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.374358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.374652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.374959 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.375694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.378015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3a099d1d-f2cb-4fba-bce1-24e877e232d7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.395645 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1326/2671] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:08 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:08.396854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.409078 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-150e0955-6f01-4228-9e3b-ee40b16a4bd9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.419079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.419985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.420268 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.420575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] PATH_INFO: `/v3/projects/a9d11ef608c449d299edacfc3272ddbd/users/cf3b538a5b8f4c14992c027f7d3ca861/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.437532 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a9d11ef608c449d299edacfc3272ddbd, user_id=cf3b538a5b8f4c14992c027f7d3ca861, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.440312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.457887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1346/2672] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:08 2022] PUT /identity/v3/projects/a9d11ef608c449d299edacfc3272ddbd/users/cf3b538a5b8f4c14992c027f7d3ca861/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 61 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:08.469721 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.470068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.470381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.498320 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] MFA Rules not processed for user `cf3b538a5b8f4c14992c027f7d3ca861`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:08.499593 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:08.566615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1327/2673] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:08 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 98 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:08.572008 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.643817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fabf5655-b3c9-4b36-8f36-5317272abe72 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.649173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.649839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.650125 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.650427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.651173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.653452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.683379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1347/2674] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:08 2022] POST /identity/v3/projects => generated 375 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:08.687468 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.699418 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f3afe9a-e727-4804-b287-9ec884623c2a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.762664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.763330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.763660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.763955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.764665 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.766952 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.787778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1328/2675] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:08 2022] POST /identity/v3/users => generated 294 bytes in 101 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:08.793326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.806260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f7f8d50a-093d-49e7-b0b8-111b4b67e177 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.811942 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.813096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.813606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.814113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.815576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.819015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.831135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1348/2676] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:08 2022] GET /identity/v3/roles => generated 1420 bytes in 39 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:08.835414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.845330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41ca218c-828f-43e8-a84f-b2c33fb0418a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.850062 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.851660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.852028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.852377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] PATH_INFO: `/v3/projects/c23653d27f5c40318ab895f70bff1182/users/814a2b0feaac4140811a7f0ff28546b6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.868580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=94,keystone:INSERT=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:08.872507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] RBAC: Authorizing `identity:create_grant(project_id=c23653d27f5c40318ab895f70bff1182, user_id=814a2b0feaac4140811a7f0ff28546b6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.876388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.892081 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1329/2677] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:08 2022] PUT /identity/v3/projects/c23653d27f5c40318ab895f70bff1182/users/814a2b0feaac4140811a7f0ff28546b6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 57 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:08.896488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.907466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-17941f14-85cf-4fba-be8a-b1d911f2c0c4 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.964025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.964695 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.964982 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.965261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:08.965984 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:08.967631 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73f20af4-52c4-4abb-9cf6-bf5b9d74c6cc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:08.978010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1349/2678] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:08 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:08.982164 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:08.992439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8a8c5521-b600-41d0-ab88-ce2ad517391d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:08.997217 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:08.998128 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:08.998434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:08.998707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] PATH_INFO: `/v3/projects/c23653d27f5c40318ab895f70bff1182/users/814a2b0feaac4140811a7f0ff28546b6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:09.000765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] RBAC: Authorizing `identity:create_grant(project_id=c23653d27f5c40318ab895f70bff1182, user_id=814a2b0feaac4140811a7f0ff28546b6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:09.003481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:09.013853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1330/2679] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:08 2022] PUT /identity/v3/projects/c23653d27f5c40318ab895f70bff1182/users/814a2b0feaac4140811a7f0ff28546b6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:09.024422 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:09.024948 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:09.025262 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:09.055322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] MFA Rules not processed for user `814a2b0feaac4140811a7f0ff28546b6`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:09.055322 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:09.070994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:09.133756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1350/2680] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:09 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 111 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:09.144635 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:09.146408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:09.206000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-186abaf7-bc4b-42f6-ac24-b54828f4bc1c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:09.211367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:09.212015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:09.212305 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:09.212583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:09.278905 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:09.280707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:09.294040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1331/2681] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:09 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 150 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:10.814149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:10.815700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:10.833902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b041f8ff-ef5e-4d42-a155-8fc77889f895 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:10.840363 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:10.841525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:10.842001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:10.842444 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:10.906549 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:10.908018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bdb1b239-1512-428a-b403-1032b921a6aa service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:10.916829 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1351/2682] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:10 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 103 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:12.820977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:12.829427 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91f73801-e1b1-4103-bef5-172e3b0ac46d service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:12.833285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:12.833920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:12.834198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:12.835778 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:12.845363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:12.846820 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:12.856955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1332/2683] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:12 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 37 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:13.628016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:13.628660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:13.628660 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:13.647094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:13.648563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:13.713875 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1352/2684] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:08:13 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:13.764111 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:13.765544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:13.837476 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d7ef201c-4b31-43da-9ba6-3c7c9017c41e service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:13.843048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:13.843716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:13.843985 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:13.844259 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:13.853411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:13.855046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bbd4b06-3407-4074-9df0-0148951828b3 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:13.863020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1333/2685] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:08:13 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:08:16.936563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:17.004770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:17.018821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6c188ac1-4f24-4667-8495-7fba5c31de8e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:17.023472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:17.024361 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:17.024645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:17.024936 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] PATH_INFO: `/v3/users/c7e7c6043fad414d89db086df799311b` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:17.026118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] RBAC: Authorizing `identity:delete_user(user_id=c7e7c6043fad414d89db086df799311b)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:17.028109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:17.121625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'c7e7c6043fad414d89db086df799311b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.131327 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'c7e7c6043fad414d89db086df799311b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.132009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'c7e7c6043fad414d89db086df799311b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.142031 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'c7e7c6043fad414d89db086df799311b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.142737 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'c7e7c6043fad414d89db086df799311b'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.167827 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1353/2686] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:16 2022] DELETE /identity/v3/users/c7e7c6043fad414d89db086df799311b => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:17.172797 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:17.173550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:17.173550 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:17.191878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:17.193787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:17.199147 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:17.275862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1334/2687] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:17 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:17.286416 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:17.287992 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:17.374345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73beab52-9511-4294-a4a4-1be0ff9edeb5 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:17.379584 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:17.380561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:17.383668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:17.384176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:17.400672 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:17.402809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:17.418687 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1354/2688] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:17 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 133 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:17.733006 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:17.734615 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:17.772216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-165e1040-b6d1-4a63-8022-fc3b9d21ffb2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:17.778304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:17.780032 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:17.780322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:17.780601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] PATH_INFO: `/v3/projects/d44bf1b03a7d4d4190165b383dec5a1c` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:17.781865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] RBAC: Authorizing `identity:delete_project(project_id=d44bf1b03a7d4d4190165b383dec5a1c)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:17.784045 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:17.844628 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'd44bf1b03a7d4d4190165b383dec5a1c', 'request_id': 'req-0fda2121-b868-4580-8562-1c21ce7b0e0e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.852188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'd44bf1b03a7d4d4190165b383dec5a1c', 'request_id': 'req-0fda2121-b868-4580-8562-1c21ce7b0e0e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.862035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'd44bf1b03a7d4d4190165b383dec5a1c', 'request_id': 'req-0fda2121-b868-4580-8562-1c21ce7b0e0e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.877508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] The token cache is being invalidate because project d44bf1b03a7d4d4190165b383dec5a1c was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:17.878012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.878979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:17.883939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1335/2689] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:17 2022] DELETE /identity/v3/projects/d44bf1b03a7d4d4190165b383dec5a1c => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:17.885127 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:17.895651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0455f31a-ce76-4506-aaed-aec60b45c9c5 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:17.961844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:17.962893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:17.963368 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:17.964021 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] PATH_INFO: `/v3/users/cf3b538a5b8f4c14992c027f7d3ca861` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:17.965438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf3b538a5b8f4c14992c027f7d3ca861)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:17.967786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:17.989224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=88,keystone:DELETE=11,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:18.033284 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf3b538a5b8f4c14992c027f7d3ca861'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.045218 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf3b538a5b8f4c14992c027f7d3ca861'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.046559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf3b538a5b8f4c14992c027f7d3ca861'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.057869 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf3b538a5b8f4c14992c027f7d3ca861'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.058582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf3b538a5b8f4c14992c027f7d3ca861'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.083978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1355/2690] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:17 2022] DELETE /identity/v3/users/cf3b538a5b8f4c14992c027f7d3ca861 => generated 0 bytes in 199 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:18.467441 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:18.470551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:18.497340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0fda2121-b868-4580-8562-1c21ce7b0e0e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:18.588117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:18.589579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:18.590010 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:18.590522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] PATH_INFO: `/v3/projects/a9d11ef608c449d299edacfc3272ddbd` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:18.592137 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] RBAC: Authorizing `identity:delete_project(project_id=a9d11ef608c449d299edacfc3272ddbd)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:18.596118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:18.650473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=9,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:18.668063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a9d11ef608c449d299edacfc3272ddbd', 'request_id': 'req-72ebc764-bb03-4c05-93bd-6652f911efe2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.675270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a9d11ef608c449d299edacfc3272ddbd', 'request_id': 'req-72ebc764-bb03-4c05-93bd-6652f911efe2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.685364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a9d11ef608c449d299edacfc3272ddbd', 'request_id': 'req-72ebc764-bb03-4c05-93bd-6652f911efe2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.701053 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] The token cache is being invalidate because project a9d11ef608c449d299edacfc3272ddbd was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:18.701522 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.702270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.704293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1336/2691] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:18 2022] DELETE /identity/v3/projects/a9d11ef608c449d299edacfc3272ddbd => generated 0 bytes in 238 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:18.708238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:18.711090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:18.726239 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7b2ef971-2ef9-4608-bd49-df1aa437627e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:18.794051 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:18.795238 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:18.795523 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:18.795801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] PATH_INFO: `/v3/users/814a2b0feaac4140811a7f0ff28546b6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:18.797142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] RBAC: Authorizing `identity:delete_user(user_id=814a2b0feaac4140811a7f0ff28546b6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:18.799151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:18.870336 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '814a2b0feaac4140811a7f0ff28546b6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.878280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '814a2b0feaac4140811a7f0ff28546b6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.879018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '814a2b0feaac4140811a7f0ff28546b6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.886986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '814a2b0feaac4140811a7f0ff28546b6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.887901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-416c48a3-bb0a-43bf-b019-2379856a96bc admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '814a2b0feaac4140811a7f0ff28546b6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:18.911673 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1356/2692] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:18 2022] DELETE /identity/v3/users/814a2b0feaac4140811a7f0ff28546b6 => generated 0 bytes in 201 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:19.203095 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.204711 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:19.231688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-72ebc764-bb03-4c05-93bd-6652f911efe2 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.298480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.299725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.300239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.300733 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] PATH_INFO: `/v3/projects/c23653d27f5c40318ab895f70bff1182` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.302275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] RBAC: Authorizing `identity:delete_project(project_id=c23653d27f5c40318ab895f70bff1182)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.305979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.360415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c23653d27f5c40318ab895f70bff1182', 'request_id': 'req-010d6fc6-91bd-451a-825c-d15227fa543e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:19.371168 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c23653d27f5c40318ab895f70bff1182', 'request_id': 'req-010d6fc6-91bd-451a-825c-d15227fa543e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:19.381046 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c23653d27f5c40318ab895f70bff1182', 'request_id': 'req-010d6fc6-91bd-451a-825c-d15227fa543e'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:19.400194 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] The token cache is being invalidate because project c23653d27f5c40318ab895f70bff1182 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:19.400533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:19.401153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-010d6fc6-91bd-451a-825c-d15227fa543e admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:19.402720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1337/2693] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:19 2022] DELETE /identity/v3/projects/c23653d27f5c40318ab895f70bff1182 => generated 0 bytes in 200 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:19.408791 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.409868 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.410612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.441215 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:19.443588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:19.455634 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=19,keystone:SELECT=79,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:19.518845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1357/2694] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:19.527481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.528019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.528019 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.547528 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:19.548840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:19.558811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1338/2695] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:19 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:19.562882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.576223 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cfdb46ef-2599-44de-8966-b8a6285e8f57 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.584094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.585674 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.586110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.586699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.587925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:19.588496 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.590804 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.608592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1358/2696] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:19 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 43 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:19.609902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.626156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-79a21cdc-4f01-40bc-b482-0b57dfa1964f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.634503 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.635769 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.636392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.636788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.637616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.640414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.681091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1339/2697] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:19 2022] POST /identity/v3/projects => generated 389 bytes in 68 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:19.682645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.698330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9b0a63f4-2188-4a28-9a84-68269e437139 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.772541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.773319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.773591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.773823 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.774515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.776717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.806003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1359/2698] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:19 2022] POST /identity/v3/users => generated 295 bytes in 121 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:19.807712 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.822742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2b0e5917-e520-4f51-9917-f6ad3b0648a1 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.828106 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.828811 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.829104 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.829390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.830149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.831853 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.844729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1340/2699] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:19 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:19.849408 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.860962 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2bd3bd26-0044-4a50-97b0-6adbbc6cbb62 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.865941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.867335 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.867617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.867893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.868641 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.870331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-14588325-83cb-47dd-8023-04580d39b631 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.884807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1360/2700] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:19 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:19.890650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:19.902075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe9aa9f9-8824-480d-bcf0-6913f4b7c25d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:19.909459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:19.910387 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.910684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.911001 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] PATH_INFO: `/v3/projects/faa838fcc1ad4ae2b8fdb778b78273bb/users/ac124c968500407b9072fe38c0553dbe/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:19.926064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:DELETE=12,keystone:SELECT=85,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:19.932479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] RBAC: Authorizing `identity:create_grant(project_id=faa838fcc1ad4ae2b8fdb778b78273bb, user_id=ac124c968500407b9072fe38c0553dbe, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:19.936096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:19.951808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1341/2701] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:19 2022] PUT /identity/v3/projects/faa838fcc1ad4ae2b8fdb778b78273bb/users/ac124c968500407b9072fe38c0553dbe/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 62 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:19.962389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:19.962830 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:19.963253 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:20.008465 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] MFA Rules not processed for user `ac124c968500407b9072fe38c0553dbe`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:20.010282 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:20.101441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1361/2702] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:19 2022] POST /identity/v3/auth/tokens => generated 3256 bytes in 140 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:20.108408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:20.110358 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:20.175688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bd11bdb5-f692-4ec7-b2bf-7e819576c422 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:20.179967 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:20.180641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:20.181040 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:20.181204 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:20.190722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:20.192188 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a96dd47b-3d19-4ceb-8cc1-5db6771c0a24 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:20.201921 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1342/2703] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:20 2022] GET /identity/v3/auth/tokens => generated 3256 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:25.960647 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:25.965379 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:25.977981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:26.081431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c4003e-234d-4816-9b16-c030c7e06f49 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:26.086858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:26.087765 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:26.088056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:26.088331 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] PATH_INFO: `/v3/users/88bd892779ce4ec38be10276c86bf9f0` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:26.089581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] RBAC: Authorizing `identity:delete_user(user_id=88bd892779ce4ec38be10276c86bf9f0)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:26.091625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:26.200203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '88bd892779ce4ec38be10276c86bf9f0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.219644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '88bd892779ce4ec38be10276c86bf9f0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.220579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '88bd892779ce4ec38be10276c86bf9f0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.231817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '88bd892779ce4ec38be10276c86bf9f0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.232873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '88bd892779ce4ec38be10276c86bf9f0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.279625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1362/2704] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:25 2022] DELETE /identity/v3/users/88bd892779ce4ec38be10276c86bf9f0 => generated 0 bytes in 320 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:26.285932 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:26.286474 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:26.286586 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:26.311416 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:26.313122 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:26.385898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1343/2705] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:26 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:26.393867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:26.395981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:26.471694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6d6789a-17f8-4098-872c-ae4e5d415044 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:26.475639 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:26.477189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:26.477468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:26.477740 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:26.488162 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:26.489857 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:26.500382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1363/2706] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:26 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:26.793352 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:26.794849 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:26.809229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a4dddbe5-da2c-471a-8daf-04cbbd3d3a67 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:26.815497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:26.816505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:26.816857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:26.817228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] PATH_INFO: `/v3/projects/ad1202a4b89646d5949a981ec1435930` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:26.818747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] RBAC: Authorizing `identity:delete_project(project_id=ad1202a4b89646d5949a981ec1435930)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:26.820928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:26.880511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'ad1202a4b89646d5949a981ec1435930', 'request_id': 'req-5112ecbb-ca84-4db1-b0b6-a51024998dbd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.887930 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'ad1202a4b89646d5949a981ec1435930', 'request_id': 'req-5112ecbb-ca84-4db1-b0b6-a51024998dbd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.896020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'ad1202a4b89646d5949a981ec1435930', 'request_id': 'req-5112ecbb-ca84-4db1-b0b6-a51024998dbd'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.910112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] The token cache is being invalidate because project ad1202a4b89646d5949a981ec1435930 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:26.910518 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.911334 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:26.914255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1344/2707] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:26 2022] DELETE /identity/v3/projects/ad1202a4b89646d5949a981ec1435930 => generated 0 bytes in 121 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:26.918582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:26.930820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3f2b08e4-deed-49af-b9aa-9e7c151fdfca service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:26.962236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:26.997939 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:26.999040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:26.999411 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:26.999788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] PATH_INFO: `/v3/users/5369d5aace8646568f0d96365429527f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.001087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] RBAC: Authorizing `identity:delete_user(user_id=5369d5aace8646568f0d96365429527f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:27.003141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:27.087756 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5369d5aace8646568f0d96365429527f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.098911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5369d5aace8646568f0d96365429527f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.101354 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5369d5aace8646568f0d96365429527f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.122614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5369d5aace8646568f0d96365429527f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.123854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-2d20eaa1-22c4-40bd-a11c-032110d49bbb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5369d5aace8646568f0d96365429527f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.150170 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1364/2708] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:26 2022] DELETE /identity/v3/users/5369d5aace8646568f0d96365429527f => generated 0 bytes in 232 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:27.499965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:27.501434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:27.518962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5112ecbb-ca84-4db1-b0b6-a51024998dbd admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:27.544393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:INSERT=2,keystone:DELETE=7 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:27.606544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:27.607890 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:27.608412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:27.608915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] PATH_INFO: `/v3/projects/3cfbe95b928d4087890aa26d75ec87a5` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.611067 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] RBAC: Authorizing `identity:delete_project(project_id=3cfbe95b928d4087890aa26d75ec87a5)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:27.613216 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:27.672510 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '3cfbe95b928d4087890aa26d75ec87a5', 'request_id': 'req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.681881 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '3cfbe95b928d4087890aa26d75ec87a5', 'request_id': 'req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.691951 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '3cfbe95b928d4087890aa26d75ec87a5', 'request_id': 'req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.707054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] The token cache is being invalidate because project 3cfbe95b928d4087890aa26d75ec87a5 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:27.707438 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.708238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-5acb6abb-d7b5-4dbb-9512-2e15e2f9ac01 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:27.710880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1345/2709] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:27 2022] DELETE /identity/v3/projects/3cfbe95b928d4087890aa26d75ec87a5 => generated 0 bytes in 211 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:27.715884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:27.716294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:27.716667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.736727 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:27.738080 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:27.821002 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1365/2710] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 100 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:27.823561 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:27.823950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:27.824328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.855886 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:27.857848 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:27.879485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1346/2711] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:27 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 57 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:27.884209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:27.901022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e4418d2-4519-403b-8c80-3a5eeba276f1 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:27.906688 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:27.907398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:27.907719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:27.908060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.908712 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:27.908969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:27.913007 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:27.927873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1366/2712] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:27 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 44 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:27.932541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:27.953636 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-585ed1ef-7410-4ae0-ac30-703cce80306f None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:27.963663 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:27.964857 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:27.965434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:27.966076 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:27.967269 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:27.969943 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.002333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1347/2713] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:27 2022] POST /identity/v3/projects => generated 363 bytes in 70 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:28.007528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.019890 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1935fd4c-4fb0-4127-9687-41598c451318 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.064184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:28.098553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.099583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.100191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.100642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.101510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.103854 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.130845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1367/2714] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:28 2022] POST /identity/v3/users => generated 282 bytes in 125 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:28.138085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.150517 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6a386e1d-8eb4-4ac9-9838-078ab753e128 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.155812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.156865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.157151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.157555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.158292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.159925 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.173052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1348/2715] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:28.179249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.191245 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8501551-bad3-490e-a4db-9f20ae8f9a3e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.197991 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.198734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.199058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.199385 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.200130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.201770 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5e326b74-8ffd-46af-be9e-7055c71cdebe admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.219484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1368/2716] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 38 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:28.221033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.234246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1ad240d4-5943-4a66-b27e-1f3d45b1d45d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.239555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4749633b-3051-487f-81af-59225c00442a admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.240773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4749633b-3051-487f-81af-59225c00442a admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.241287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4749633b-3051-487f-81af-59225c00442a admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.241910 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4749633b-3051-487f-81af-59225c00442a admin admin] PATH_INFO: `/v3/projects/0958a6432876442c91c2971f06953c67/users/7dd51c6fcb4149a0922f6e7286324a01/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.263507 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4749633b-3051-487f-81af-59225c00442a admin admin] RBAC: Authorizing `identity:create_grant(project_id=0958a6432876442c91c2971f06953c67, user_id=7dd51c6fcb4149a0922f6e7286324a01, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.266784 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4749633b-3051-487f-81af-59225c00442a admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.294800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1349/2717] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:28 2022] PUT /identity/v3/projects/0958a6432876442c91c2971f06953c67/users/7dd51c6fcb4149a0922f6e7286324a01/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 72 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:28.304200 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.304754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.305116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.346343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] MFA Rules not processed for user `7dd51c6fcb4149a0922f6e7286324a01`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:28.347608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:28.469404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1369/2718] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3230 bytes in 167 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:28.473705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4749633b-3051-487f-81af-59225c00442a admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.523922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:28.546388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4749633b-3051-487f-81af-59225c00442a admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.551374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.552061 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.552350 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.552631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.553397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.555392 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.585463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1350/2719] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:28 2022] POST /identity/v3/projects => generated 361 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:28.590394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.602809 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cff9242-9081-4040-b39d-cf126216b803 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.674194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.674922 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.675212 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.675491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.676276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.678728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.703360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1370/2720] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:28 2022] POST /identity/v3/users => generated 287 bytes in 113 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:28.707311 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.718625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7872ebfb-d934-4630-a99f-be3be10c5013 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.723708 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.724381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.724757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.725688 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.726762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.728459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.739408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1351/2721] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:28.758658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.771883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e246f112-8299-47b0-91e1-a4ddf93bf5f0 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.785702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.786611 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.786970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.787248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] PATH_INFO: `/v3/projects/67a9d093519c4089afd78170313d3143/users/8c293d0f7b2345d3a237638a618dd0d6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.804812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67a9d093519c4089afd78170313d3143, user_id=8c293d0f7b2345d3a237638a618dd0d6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.807531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.822871 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1371/2722] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:28 2022] PUT /identity/v3/projects/67a9d093519c4089afd78170313d3143/users/8c293d0f7b2345d3a237638a618dd0d6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 66 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:28.826736 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.837297 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21f16d78-7b89-4fed-87f6-51653d590467 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.908179 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.908891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.909165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.909445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.910213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.911908 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2e5a962f-6e2c-4a5c-bfef-d0eb88e051e3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.924483 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1352/2723] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:28 2022] GET /identity/v3/roles => generated 1420 bytes in 98 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:28.930177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:28.941070 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f29baf54-6bd5-4599-97a9-07a10b990789 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:28.946865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:28.947784 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.948075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.948425 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] PATH_INFO: `/v3/projects/67a9d093519c4089afd78170313d3143/users/8c293d0f7b2345d3a237638a618dd0d6/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:28.950649 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] RBAC: Authorizing `identity:create_grant(project_id=67a9d093519c4089afd78170313d3143, user_id=8c293d0f7b2345d3a237638a618dd0d6, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:28.953668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:28.961307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:INSERT=8 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:28.970559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1372/2724] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:28 2022] PUT /identity/v3/projects/67a9d093519c4089afd78170313d3143/users/8c293d0f7b2345d3a237638a618dd0d6/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 41 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:28.982605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:28.983225 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:28.983542 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:29.017356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] MFA Rules not processed for user `8c293d0f7b2345d3a237638a618dd0d6`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:29.018572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:29.095634 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1353/2725] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:28 2022] POST /identity/v3/auth/tokens => generated 3295 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:29.108113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:29.110014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:29.173821 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0aa43fe0-8165-4927-b7f3-611b2302b023 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:29.179205 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:29.179866 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:29.180176 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:29.180462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:29.191135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:29.193090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-23286e67-b26c-414f-86e6-a5efc2bda80d service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:29.206919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1373/2726] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:29 2022] GET /identity/v3/auth/tokens => generated 3295 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:29.283044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:29.296936 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2cbdbdf5-53ef-45a4-87fc-6f11a09ccb05 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:29.300761 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:29.301411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:29.301697 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:29.301971 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:29.354219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:29.388625 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:29.390348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:29.408555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1354/2727] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:29 2022] GET /identity/v3/auth/tokens => generated 3230 bytes in 126 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:30.466456 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:30.466985 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:30.467074 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:30.485575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:30.486799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:30.549109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1374/2728] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:08:30 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 84 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:30.563960 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:30.565713 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:30.667859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a09589cb-a0a7-4f66-8c7d-db14c841f7df service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:30.673252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:30.673964 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:30.674251 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:30.674596 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:30.683592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:30.685521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e0249c8e-c970-4903-99a6-7361a166f97b service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:30.694616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1355/2729] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:08:30 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 131 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 01:08:34.262678 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:34.377757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0d92beb9-c3f4-4093-8ead-3775987261cc None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:34.383627 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:34.384610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:34.384898 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:34.385177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] PATH_INFO: `/v3/users/7dd51c6fcb4149a0922f6e7286324a01` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:34.386393 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7dd51c6fcb4149a0922f6e7286324a01)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:34.388383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:34.393471 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=100 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:34.454960 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7dd51c6fcb4149a0922f6e7286324a01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.463056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7dd51c6fcb4149a0922f6e7286324a01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.463834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7dd51c6fcb4149a0922f6e7286324a01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.471059 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7dd51c6fcb4149a0922f6e7286324a01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.471665 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7dd51c6fcb4149a0922f6e7286324a01'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.489633 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1375/2730] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:34 2022] DELETE /identity/v3/users/7dd51c6fcb4149a0922f6e7286324a01 => generated 0 bytes in 228 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:34.494782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:34.495298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:34.495415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:34.543610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:34.544826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:34.605315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1356/2731] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:34 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 112 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:34.615081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:34.617149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:34.694878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f68b3c9b-81ab-4839-9d73-ec0027246125 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:34.698498 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:34.699194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:34.699484 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:34.699768 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:34.709519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:34.711054 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:34.721320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1376/2732] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:34 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 107 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:34.754652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:34.756177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:34.769388 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b132a564-c7c7-440b-b12f-8929fad61047 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:34.774741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:34.775809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:34.776162 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:34.776451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] PATH_INFO: `/v3/users/ac64444a4b794026a5e95584d31f1c04` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:34.777766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac64444a4b794026a5e95584d31f1c04)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:34.779860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:34.835230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:34.861330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac64444a4b794026a5e95584d31f1c04'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.872918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac64444a4b794026a5e95584d31f1c04'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.874149 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac64444a4b794026a5e95584d31f1c04'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.885969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac64444a4b794026a5e95584d31f1c04'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.899534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac64444a4b794026a5e95584d31f1c04'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:34.906022 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1357/2733] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:34 2022] DELETE /identity/v3/users/ac64444a4b794026a5e95584d31f1c04 => generated 0 bytes in 152 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:35.139575 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.145511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.164352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c0393bc-11c0-40f7-b4dd-1fa5f7a9e2a1 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.174670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.176410 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.252065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.253103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.253606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.253881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] PATH_INFO: `/v3/projects/0958a6432876442c91c2971f06953c67` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.255278 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=0958a6432876442c91c2971f06953c67)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.258242 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.271118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:35.277616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-66e666e5-9747-462e-a90e-e3fef327909d admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.282806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.283684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.283963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.284243 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] PATH_INFO: `/v3/projects/54b31a874a6b4ec48f9a938d817c4637` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.285485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] RBAC: Authorizing `identity:delete_project(project_id=54b31a874a6b4ec48f9a938d817c4637)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.288101 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.316482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '0958a6432876442c91c2971f06953c67', 'request_id': 'req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.325095 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '0958a6432876442c91c2971f06953c67', 'request_id': 'req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.330763 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '0958a6432876442c91c2971f06953c67', 'request_id': 'req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.343582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] The token cache is being invalidate because project 0958a6432876442c91c2971f06953c67 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:35.343883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.344536 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.346469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1377/2734] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:35 2022] DELETE /identity/v3/projects/0958a6432876442c91c2971f06953c67 => generated 0 bytes in 207 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:35.350837 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.352463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.376963 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '54b31a874a6b4ec48f9a938d817c4637', 'request_id': 'req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.383142 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '54b31a874a6b4ec48f9a938d817c4637', 'request_id': 'req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.393871 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '54b31a874a6b4ec48f9a938d817c4637', 'request_id': 'req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.413704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] The token cache is being invalidate because project 54b31a874a6b4ec48f9a938d817c4637 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:35.414103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.414924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-f5d5031c-6c68-42ab-bcb7-dc13dfacf7d9 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.416865 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1358/2735] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:35 2022] DELETE /identity/v3/projects/54b31a874a6b4ec48f9a938d817c4637 => generated 0 bytes in 243 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:35.423059 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827fd7c-8bb3-4b25-a89c-e124ce129644 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.423861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827fd7c-8bb3-4b25-a89c-e124ce129644 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.424255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0827fd7c-8bb3-4b25-a89c-e124ce129644 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.433161 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ca4be618-cb59-4c06-8b7e-8d04430e0fc5 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.486924 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0827fd7c-8bb3-4b25-a89c-e124ce129644 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:35.488048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0827fd7c-8bb3-4b25-a89c-e124ce129644 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.498619 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.499534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.499817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.500094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] PATH_INFO: `/v3/users/8c293d0f7b2345d3a237638a618dd0d6` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.501312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] RBAC: Authorizing `identity:delete_user(user_id=8c293d0f7b2345d3a237638a618dd0d6)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.503261 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.514843 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=86,keystone:DELETE=12,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:35.559436 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '8c293d0f7b2345d3a237638a618dd0d6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.562636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=84,keystone:DELETE=14,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:35.570036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '8c293d0f7b2345d3a237638a618dd0d6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.571479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '8c293d0f7b2345d3a237638a618dd0d6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.576861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1359/2736] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:35.579087 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '8c293d0f7b2345d3a237638a618dd0d6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.579755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '8c293d0f7b2345d3a237638a618dd0d6'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.585237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.585720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.586002 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.598679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1378/2737] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:35 2022] DELETE /identity/v3/users/8c293d0f7b2345d3a237638a618dd0d6 => generated 0 bytes in 249 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:35.604868 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:35.606064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.671576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1360/2738] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:35.676583 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.688533 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2c0a355-5bba-46c9-ad72-8af1ef3dbc67 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.694696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.695540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.695973 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.696264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.697134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:35.697488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.699545 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.713203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1379/2739] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:35 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:08:35.717819 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.728795 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cb71651e-4776-4caa-8e54-f0df943eeb28 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.733702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.734409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.734677 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.734991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.735675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.737743 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.781568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1361/2740] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:35 2022] POST /identity/v3/projects => generated 375 bytes in 64 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:35.782349 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.794696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-bc80e853-f9b2-473b-b615-ed125983c570 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.861621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.862211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.862568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.863018 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.863796 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.866812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.894040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1380/2741] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:35 2022] POST /identity/v3/users => generated 288 bytes in 112 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:35.898746 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.901986 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.907454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:35.916450 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c486f78f-ab32-4db6-bf4f-2445a51272f5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.919998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c8377e6c-e140-418a-9271-3b92ec5e96e3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.925543 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.926420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.927038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.927038 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] PATH_INFO: `/v3/projects/67a9d093519c4089afd78170313d3143` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.928810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] RBAC: Authorizing `identity:delete_project(project_id=67a9d093519c4089afd78170313d3143)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.930887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.931670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.932463 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.932706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.932923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.934363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.935990 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.949496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1362/2742] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 51 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:35.953447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.964140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81c52915-31eb-40ca-b16e-147f3fd1dc96 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:35.969203 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:35.969873 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:35.970161 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:35.970439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:35.971234 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:35.972807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:35.985933 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '67a9d093519c4089afd78170313d3143', 'request_id': 'req-fe851236-538f-41a1-ae7e-9a7dfb40bd22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:35.986915 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1363/2743] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:35 2022] GET /identity/v3/roles => generated 1420 bytes in 34 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:35.987897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:35.996334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '67a9d093519c4089afd78170313d3143', 'request_id': 'req-fe851236-538f-41a1-ae7e-9a7dfb40bd22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:36.003845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '67a9d093519c4089afd78170313d3143', 'request_id': 'req-fe851236-538f-41a1-ae7e-9a7dfb40bd22'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:36.004779 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-78e41e21-0a4b-4692-a171-779f4f476beb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.020912 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] The token cache is being invalidate because project 67a9d093519c4089afd78170313d3143 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:36.021191 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:36.022216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fe851236-538f-41a1-ae7e-9a7dfb40bd22 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:36.024005 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1381/2744] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:35 2022] DELETE /identity/v3/projects/67a9d093519c4089afd78170313d3143 => generated 0 bytes in 122 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:36.028774 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2125f92-4747-49f2-bc19-a42f8dfd85ed None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.029184 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2125f92-4747-49f2-bc19-a42f8dfd85ed None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.029468 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2125f92-4747-49f2-bc19-a42f8dfd85ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.047954 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-a2125f92-4747-49f2-bc19-a42f8dfd85ed None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:36.049149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a2125f92-4747-49f2-bc19-a42f8dfd85ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:36.073067 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:DELETE=10,keystone:SELECT=86,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:36.079132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.080027 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.080314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.080889 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] PATH_INFO: `/v3/projects/affbcb4611414dcba4e3eca7c0686be7/users/4707858b7ef84a7fbd74a93b91cc6e68/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.099861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] RBAC: Authorizing `identity:create_grant(project_id=affbcb4611414dcba4e3eca7c0686be7, user_id=4707858b7ef84a7fbd74a93b91cc6e68, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.102638 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aa880901-b335-4e52-8701-73e0df7b1045 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.111140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:36.118123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1364/2745] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:35 2022] PUT /identity/v3/projects/affbcb4611414dcba4e3eca7c0686be7/users/4707858b7ef84a7fbd74a93b91cc6e68/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 131 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:36.122220 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1382/2746] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 94 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:36.128477 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.128773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.129048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.129538 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.129834 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.130115 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.148591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:36.149801 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:36.155583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] MFA Rules not processed for user `4707858b7ef84a7fbd74a93b91cc6e68`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:36.156757 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:36.211181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1383/2747] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:36.215149 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.223602 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1365/2748] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 96 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:36.228116 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-18165833-e801-4e0f-b532-f6ba004b6f52 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.229403 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.230058 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.230347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.230635 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.231382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:36.231743 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.233435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.235670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.237555 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:36.249119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1384/2749] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:08:36.250209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.260773 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-aac0986e-2bc6-4d0f-ab84-fefec386ddb9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.265300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.265932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.266211 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.266482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.267209 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.269097 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.299855 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1385/2750] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/projects => generated 375 bytes in 47 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:36.300482 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ff53bacc-0bfd-445c-922c-c87523453ad2 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.300843 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.346409 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.347088 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.347373 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.347648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.364194 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a6b11086-4481-486b-a17f-e8ca3ff0ebe6 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.369518 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.370195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.370482 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.370764 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.371476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.373420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.395565 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1386/2751] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/users => generated 288 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:36.399609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.409287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b5fbd539-e649-456f-9a80-5a17a1895130 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.412750 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.414319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.414686 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.415375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.415668 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.415935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.416661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.418132 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:36.418260 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.424689 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1366/2752] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 190 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:08:36.429077 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1387/2753] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:36.432874 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.443812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f2257569-acb2-4378-a48f-0e50bf05a4b8 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.448495 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.449156 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.449485 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.449763 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.450535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.452151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cc5a05f3-44e1-4041-a717-62e43c4cb641 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.462151 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1367/2754] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:36.466306 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.476799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-df772333-ffb2-4d33-8d15-aa1d17dc8c0e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.481459 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.482333 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.482616 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.482925 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] PATH_INFO: `/v3/projects/05cfb442ccbb4204ae2e3c2882f94c94/users/5a64440852794374a2e1481eec55ef65/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.492667 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:36.499849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] RBAC: Authorizing `identity:create_grant(project_id=05cfb442ccbb4204ae2e3c2882f94c94, user_id=5a64440852794374a2e1481eec55ef65, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.502754 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.517351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1388/2755] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:36 2022] PUT /identity/v3/projects/05cfb442ccbb4204ae2e3c2882f94c94/users/5a64440852794374a2e1481eec55ef65/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:36.528412 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.528972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.528972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.555630 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] MFA Rules not processed for user `5a64440852794374a2e1481eec55ef65`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:36.556840 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:36.653911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1368/2756] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 123 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:36.657516 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.736867 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d0cfc684-ac7d-4941-84a6-da4fe1212019 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.742382 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.743081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.743365 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.743640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.744330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.746244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.774181 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1389/2757] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/projects => generated 375 bytes in 119 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:36.778288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.792312 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-efc65030-33d7-4637-a491-13a12d7a9258 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.849731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.850402 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.850687 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.850998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.851694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.853652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.877825 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1369/2758] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:36 2022] POST /identity/v3/users => generated 294 bytes in 100 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:36.879164 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.892728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95bc4780-3d8b-436f-b92a-ad045b4864da admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.898094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.899066 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.899479 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.899884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.900955 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.903344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.914053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1390/2759] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:08:36.918382 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.928765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-48757a02-351e-4e49-ad15-144055f6acc6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:36.933363 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:36.934239 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:36.934524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:36.934835 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] PATH_INFO: `/v3/projects/cb68b3b3c49f4743bcde95b167282e11/users/5bfcbd09720d4e90ade54e4a77f2baa1/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:36.951299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb68b3b3c49f4743bcde95b167282e11, user_id=5bfcbd09720d4e90ade54e4a77f2baa1, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:36.954005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:36.967682 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1370/2760] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:36 2022] PUT /identity/v3/projects/cb68b3b3c49f4743bcde95b167282e11/users/5bfcbd09720d4e90ade54e4a77f2baa1/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 50 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:36.971721 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:36.982298 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fa27409a-3889-4075-a9b9-6f9e94a1c962 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:37.042477 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:37.043190 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:37.043561 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:37.043839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:37.044734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:37.046654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2f2e572b-3a8b-4f9a-b25c-5bfd1e4fcae2 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:37.057845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1391/2761] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:36 2022] GET /identity/v3/roles => generated 1420 bytes in 87 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:37.062072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:37.073199 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffe74c2e-9fcc-4ef8-ba0c-b52def046ee2 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:37.077866 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:37.078790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:37.079085 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:37.079384 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] PATH_INFO: `/v3/projects/cb68b3b3c49f4743bcde95b167282e11/users/5bfcbd09720d4e90ade54e4a77f2baa1/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:37.081359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] RBAC: Authorizing `identity:create_grant(project_id=cb68b3b3c49f4743bcde95b167282e11, user_id=5bfcbd09720d4e90ade54e4a77f2baa1, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:37.084056 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:37.094877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1371/2762] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:37 2022] PUT /identity/v3/projects/cb68b3b3c49f4743bcde95b167282e11/users/5bfcbd09720d4e90ade54e4a77f2baa1/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 33 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:37.160914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fe130-e7b7-4560-b2a2-2bc08349d9b1 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:37.161173 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fe130-e7b7-4560-b2a2-2bc08349d9b1 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:37.161460 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-310fe130-e7b7-4560-b2a2-2bc08349d9b1 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:37.192845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-310fe130-e7b7-4560-b2a2-2bc08349d9b1 None None] MFA Rules not processed for user `5bfcbd09720d4e90ade54e4a77f2baa1`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:37.194165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-310fe130-e7b7-4560-b2a2-2bc08349d9b1 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:37.237313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:37.262311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1392/2763] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:37 2022] POST /identity/v3/auth/tokens => generated 3309 bytes in 103 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:37.273533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:37.306055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:37.331678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-51f4042a-f91e-42ea-97b8-6774692e3dec admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:37.335405 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:37.336044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:37.336327 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:37.336601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:37.392184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:37.393622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:37.403508 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1372/2764] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:37 2022] GET /identity/v3/auth/tokens => generated 3242 bytes in 130 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:37.806685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef119e-6650-44b8-8935-cabd5bddc09c None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:37.807257 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef119e-6650-44b8-8935-cabd5bddc09c None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:37.807362 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9eef119e-6650-44b8-8935-cabd5bddc09c None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:37.825249 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-9eef119e-6650-44b8-8935-cabd5bddc09c None None] MFA Rules not processed for user `4707858b7ef84a7fbd74a93b91cc6e68`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:37.826451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9eef119e-6650-44b8-8935-cabd5bddc09c None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:37.880316 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1393/2765] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:37 2022] POST /identity/v3/auth/tokens => generated 3242 bytes in 75 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:08:39.956502 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:39.957920 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.022268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a970d41-624c-4d47-820d-9a7ff4823220 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.026740 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.027623 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.027902 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.028174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] PATH_INFO: `/v3/users/ac124c968500407b9072fe38c0553dbe` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.029351 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] RBAC: Authorizing `identity:delete_user(user_id=ac124c968500407b9072fe38c0553dbe)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.031291 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:40.076903 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'ac124c968500407b9072fe38c0553dbe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.084318 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'ac124c968500407b9072fe38c0553dbe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.085082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'ac124c968500407b9072fe38c0553dbe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.092282 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'ac124c968500407b9072fe38c0553dbe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.104575 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'ac124c968500407b9072fe38c0553dbe'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.110459 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1373/2766] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:08:39 2022] DELETE /identity/v3/users/ac124c968500407b9072fe38c0553dbe => generated 0 bytes in 154 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:40.115003 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.116233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.116233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.133252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:40.134582 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.191593 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1394/2767] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 78 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:40.201678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:40.203116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.208511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:40.263228 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-73a406be-d3d7-4887-a8f1-9181d02a5846 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.266895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.267539 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.267826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.268107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.277805 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.279852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c25d5552-a23c-456e-ada5-d77ea109823a service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:40.290862 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1374/2768] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:40 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 90 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:40.561233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:40.562658 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.575420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dcdca785-8000-4397-a0e4-71413417a67b None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.580192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.581281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.581625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.581963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] PATH_INFO: `/v3/projects/faa838fcc1ad4ae2b8fdb778b78273bb` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.583250 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] RBAC: Authorizing `identity:delete_project(project_id=faa838fcc1ad4ae2b8fdb778b78273bb)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.586182 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:40.634897 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'faa838fcc1ad4ae2b8fdb778b78273bb', 'request_id': 'req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.642178 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'faa838fcc1ad4ae2b8fdb778b78273bb', 'request_id': 'req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.647980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'faa838fcc1ad4ae2b8fdb778b78273bb', 'request_id': 'req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.658742 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] The token cache is being invalidate because project faa838fcc1ad4ae2b8fdb778b78273bb was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:08:40.659053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.659684 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-55892bc1-c67e-4c15-b68b-9eb7ae2c7b94 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:08:40.661221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1395/2769] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:08:40 2022] DELETE /identity/v3/projects/faa838fcc1ad4ae2b8fdb778b78273bb => generated 0 bytes in 101 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:40.666479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.666798 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.667073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.684878 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:40.686082 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.744793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1375/2770] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 79 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:40.807606 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.807876 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.808147 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.826472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:40.827703 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:40.835729 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1396/2771] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:40 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:40.840198 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:40.850304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e4940898-0dd4-4d0e-a2ed-a0f97f3400c8 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.854861 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.855509 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.855793 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.856068 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.856780 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:08:40.857102 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.858669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:40.871004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1376/2772] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:08:40 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 177 bytes (2 switches on core 0) Jul 24 01:08:40.875395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:40.881101 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:40.885022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e27f7103-eeaa-453a-a110-34115b45c2da None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.891189 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.891851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.892348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.892799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.893493 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.895464 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:40.921659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1397/2773] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:40 2022] POST /identity/v3/projects => generated 377 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:40.925288 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:40.935818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dd3ddfb2-367f-46f6-8015-8f75b0e190be admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:40.993336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:40.994012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:40.994299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:40.994576 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:40.995313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:40.997229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:41.016403 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1377/2774] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:40 2022] POST /identity/v3/users => generated 289 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:08:41.020846 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:41.033534 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-143e4f40-bd79-49df-b7d2-fb7bacf49aa3 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:41.041056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:41.041725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:41.042009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:41.042285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:41.043044 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:41.044643 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:41.056571 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1398/2775] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 36 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:41.060807 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:41.071064 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f6f2ef9e-f47a-477e-a791-c4b4d8ca5fdb admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:41.075583 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:41.076229 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:41.076512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:41.076788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:41.077501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:41.079145 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-301c9229-c1b1-49ce-827b-4bcf8bf449ff admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:41.088969 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1378/2776] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:41 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:41.092990 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:41.103698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7ece978f-2968-4a49-9cca-e38ffec81c8e admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:41.108276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:41.109145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:41.109427 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:41.109702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] PATH_INFO: `/v3/projects/be64d6ea0f0149608c9a823425a0e811/users/7c545f7be7794566b639c29c0717c17a/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:41.124591 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] RBAC: Authorizing `identity:create_grant(project_id=be64d6ea0f0149608c9a823425a0e811, user_id=7c545f7be7794566b639c29c0717c17a, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:41.127293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:41.140085 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1399/2777] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:41 2022] PUT /identity/v3/projects/be64d6ea0f0149608c9a823425a0e811/users/7c545f7be7794566b639c29c0717c17a/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 47 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:41.144989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:41.145286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:41.145570 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:41.164270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:41.165473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:41.170375 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:41.225551 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1379/2778] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:41 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 82 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:41.235273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:41.236957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:41.293134 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3bea443e-913f-48ad-9a89-03e27d74988d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:41.296815 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:41.297463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:41.297748 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:41.298024 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:41.307022 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:41.308466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5a89daa4-ef03-431e-99d1-d60db1af2c88 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:41.318312 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1400/2779] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:41 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (2 switches on core 0) Jul 24 01:08:42.023126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:42.024601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:42.084775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-7a3e5c08-22c2-48d2-8bbd-5405530b8a6a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:42.088408 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:42.089058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:42.089341 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:42.089616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:42.091473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:42.161869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:42.164330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:42.175436 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1380/2780] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:42 2022] GET /identity/v3/auth/tokens => generated 3309 bytes in 154 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:08:46.198269 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.198835 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.198977 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.234210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] MFA Rules not processed for user `7c545f7be7794566b639c29c0717c17a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:46.235710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:46.321106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1401/2781] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:46 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 124 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:46.325514 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.337548 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a3433a6c-f64b-4f68-92b6-ac6391a91a10 service cinder] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.356299 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.357611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.358206 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.358800 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.360322 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.362818 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.431359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1381/2782] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:08:46 2022] POST /identity/v3/projects => generated 379 bytes in 106 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:46.431576 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.446108 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c19bcd74-1e02-4ba1-896f-18e98929347e None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.500221 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=98,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:46.529171 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.529858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.530143 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.530420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.531139 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.533124 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.576385 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.583032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1402/2783] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:08:46 2022] POST /identity/v3/users => generated 296 bytes in 152 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:08:46.592092 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d6647c4-eb24-4adb-9b88-cb40ad1186ca admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.600411 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.601540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.602159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.602661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.604667 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.606566 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.628786 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.633753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1382/2784] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 58 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:46.641131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b7357a9f-3214-45e5-b5c5-b186e10caabe admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.646716 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.647657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.647935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.648208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] PATH_INFO: `/v3/projects/c41a498067354c0a81f56e90db1da183/users/a8c1e286d5ad40dc89ffb9baca32925f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.665638 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41a498067354c0a81f56e90db1da183, user_id=a8c1e286d5ad40dc89ffb9baca32925f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.668404 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.686360 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1403/2785] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:46 2022] PUT /identity/v3/projects/c41a498067354c0a81f56e90db1da183/users/a8c1e286d5ad40dc89ffb9baca32925f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 58 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:08:46.690131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.702220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f1634119-1e55-4402-880a-f1e7cfb2f384 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.733802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:08:46.766735 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.767443 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.767727 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.768005 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.768728 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.770331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-26c535da-7ea1-49b2-8062-698a0c60e877 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.783458 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1383/2786] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:08:46 2022] GET /identity/v3/roles => generated 1420 bytes in 91 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:08:46.784873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:46.795787 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0bd7ed79-518f-451a-9fa9-45d0dc7ed7e7 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:46.800617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:46.801499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:46.801780 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:46.802056 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] PATH_INFO: `/v3/projects/c41a498067354c0a81f56e90db1da183/users/a8c1e286d5ad40dc89ffb9baca32925f/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:46.804799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] RBAC: Authorizing `identity:create_grant(project_id=c41a498067354c0a81f56e90db1da183, user_id=a8c1e286d5ad40dc89ffb9baca32925f, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:46.807514 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:46.818798 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1404/2787] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:08:46 2022] PUT /identity/v3/projects/c41a498067354c0a81f56e90db1da183/users/a8c1e286d5ad40dc89ffb9baca32925f/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 34 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:08:52.330330 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:52.330806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:52.330919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:52.363776 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] MFA Rules not processed for user `a8c1e286d5ad40dc89ffb9baca32925f`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:08:52.365643 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:52.458639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1384/2788] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:08:52 2022] POST /identity/v3/auth/tokens => generated 3313 bytes in 129 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:08:52.473386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:08:52.475491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:08:52.629969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e528ead1-cd00-460d-bf72-b32fc4a05e83 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:08:52.636677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:08:52.637334 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:08:52.637617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:08:52.637893 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:08:52.705816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:08:52.707451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-91cce270-d348-4972-91b4-c8599d9c905d service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:08:52.717941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1405/2789] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:08:52 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 245 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:02.454622 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=50 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:02.713621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=94,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:03.696075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:03.697498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:03.790621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6cca3689-5703-4451-91fa-99eec1a41e6a None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:03.796544 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:03.797255 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:03.797552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:03.797882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:03.799791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:03.812131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:03.813720 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:03.825042 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1385/2790] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:03 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 129 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:04.784113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60ed25f-3e25-412f-b6f3-e13d79cb4a47 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:04.784659 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60ed25f-3e25-412f-b6f3-e13d79cb4a47 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:04.785167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d60ed25f-3e25-412f-b6f3-e13d79cb4a47 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:04.817628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-d60ed25f-3e25-412f-b6f3-e13d79cb4a47 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:04.818849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-d60ed25f-3e25-412f-b6f3-e13d79cb4a47 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:04.873431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1406/2791] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:09:04 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 91 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:04.885569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:04.887012 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:04.959193 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b32373ec-8bd4-4845-91b8-7e2ca1addaa2 service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:04.965210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:04.966075 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:04.966467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:04.966754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:04.975782 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:04.977365 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:04.993301 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1386/2792] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:09:04 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 01:09:05.506358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.506646 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.506957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.526600 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:05.527910 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:05.535113 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1407/2793] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:05.548220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:05.559974 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-562e1c47-ceeb-4bff-a80d-7805c453e566 service swift] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:05.565449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:05.566236 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.566525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.567449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.576241 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:05.577823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5878e4f8-ed88-48bc-95c1-22cbd4b0056a service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:05.585824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1387/2794] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:09:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 38 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:09:05.777849 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:05.778973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.779263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.779494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:05.779598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.798804 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:05.800037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:05.807439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1388/2795] 10.222.0.18 () {58 vars in 1009 bytes} [Sun Jul 24 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 29 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:09:05.819911 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:05.834753 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4cf8feb2-ac30-4069-886b-473f2e0c11cd None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:05.840772 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:05.841422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.841709 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.841986 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.852347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c41258f1-4cd8-40d7-a932-c17a9e171cf6 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:05.853048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:05.854534 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1631946b-940f-478c-bedf-9310f81799ce service swift] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:05.857511 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:05.858494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.858783 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.859075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] PATH_INFO: `/v3/users/5a64440852794374a2e1481eec55ef65` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.860541 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] RBAC: Authorizing `identity:delete_user(user_id=5a64440852794374a2e1481eec55ef65)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:05.862651 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:05.868444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1389/2796] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:09:05 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 49 msecs (HTTP/1.1 200) 6 headers in 379 bytes (2 switches on core 0) Jul 24 01:09:05.920319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5a64440852794374a2e1481eec55ef65'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:05.927926 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5a64440852794374a2e1481eec55ef65'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:05.928551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5a64440852794374a2e1481eec55ef65'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:05.937130 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5a64440852794374a2e1481eec55ef65'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:05.938096 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5a64440852794374a2e1481eec55ef65'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:05.959865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1408/2797] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:05 2022] DELETE /identity/v3/users/5a64440852794374a2e1481eec55ef65 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:05.964725 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9230d06d-d633-4f23-bc37-28a89cdad4e7 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:05.965366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9230d06d-d633-4f23-bc37-28a89cdad4e7 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:05.965366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9230d06d-d633-4f23-bc37-28a89cdad4e7 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:05.989192 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-9230d06d-d633-4f23-bc37-28a89cdad4e7 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:05.990546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9230d06d-d633-4f23-bc37-28a89cdad4e7 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:06.023324 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:06.077254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1390/2798] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:05 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 114 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:06.088475 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:06.089972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:06.106001 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:06.187730 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-590169d8-3c10-4612-95bb-a3f88c14ef3d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:06.193677 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:06.194339 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:06.194624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:06.194937 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:06.205902 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:06.207442 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:06.219009 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1409/2799] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:06 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 132 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:06.252353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:06.252724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:06.253072 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:06.273196 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] MFA Rules not processed for user `f2b854f126714f82bfa98223c38e7376`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:06.274694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:06.338276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1391/2800] 10.222.0.18 () {58 vars in 967 bytes} [Sun Jul 24 01:09:06 2022] POST /identity/v3/auth/tokens => generated 3185 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:06.351685 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:06.353232 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:06.441418 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0895e0e1-379b-49a4-8e24-0aff9a1f15bf service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:06.448589 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'd1387a119eb04d0a9efe0164de9a8c3c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:06.449276 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:06.449570 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:06.449848 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:06.458988 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:06.460947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:06.470728 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1410/2801] 10.222.0.18 () {60 vars in 1384 bytes} [Sun Jul 24 01:09:06 2022] GET /identity/v3/auth/tokens?nocatalog => generated 604 bytes in 120 msecs (HTTP/1.1 200) 6 headers in 379 bytes (1 switches on core 0) Jul 24 01:09:06.519955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:06.521616 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:06.537230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a19593ac-b7f6-4126-ae44-91f85bf7545d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:06.542219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:06.543181 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:06.543660 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:06.544123 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] PATH_INFO: `/v3/projects/05cfb442ccbb4204ae2e3c2882f94c94` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:06.545367 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] RBAC: Authorizing `identity:delete_project(project_id=05cfb442ccbb4204ae2e3c2882f94c94)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:06.547573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:06.601826 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '05cfb442ccbb4204ae2e3c2882f94c94', 'request_id': 'req-84e4853d-eb11-42bf-8502-ea24aed02bdf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.612426 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '05cfb442ccbb4204ae2e3c2882f94c94', 'request_id': 'req-84e4853d-eb11-42bf-8502-ea24aed02bdf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.622860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '05cfb442ccbb4204ae2e3c2882f94c94', 'request_id': 'req-84e4853d-eb11-42bf-8502-ea24aed02bdf'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.641836 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] The token cache is being invalidate because project 05cfb442ccbb4204ae2e3c2882f94c94 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:06.642131 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.642859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.644648 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1392/2802] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:06 2022] DELETE /identity/v3/projects/05cfb442ccbb4204ae2e3c2882f94c94 => generated 0 bytes in 126 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:06.649107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:06.660994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-defd9727-2463-44a0-a6d1-a145c6bd1218 service swift] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:06.725237 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:06.726273 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:06.726504 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:06.726794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] PATH_INFO: `/v3/users/5bfcbd09720d4e90ade54e4a77f2baa1` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:06.728036 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] RBAC: Authorizing `identity:delete_user(user_id=5bfcbd09720d4e90ade54e4a77f2baa1)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:06.730252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:06.756877 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=96,keystone:DELETE=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:06.782525 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '5bfcbd09720d4e90ade54e4a77f2baa1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.795129 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '5bfcbd09720d4e90ade54e4a77f2baa1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.795799 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '5bfcbd09720d4e90ade54e4a77f2baa1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.803521 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '5bfcbd09720d4e90ade54e4a77f2baa1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.804399 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1307b4c7-d5fa-44ff-b079-c6a519b94c12 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '5bfcbd09720d4e90ade54e4a77f2baa1'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:06.825793 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1411/2803] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:06 2022] DELETE /identity/v3/users/5bfcbd09720d4e90ade54e4a77f2baa1 => generated 0 bytes in 177 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:07.122702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.124698 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:07.140834 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84e4853d-eb11-42bf-8502-ea24aed02bdf admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.206999 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:07.216304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.217328 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.217610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.217891 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] PATH_INFO: `/v3/projects/cb68b3b3c49f4743bcde95b167282e11` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.219372 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] RBAC: Authorizing `identity:delete_project(project_id=cb68b3b3c49f4743bcde95b167282e11)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.221957 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.273722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'cb68b3b3c49f4743bcde95b167282e11', 'request_id': 'req-a8f47591-6137-4ff4-8679-0f31e8e9a66c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:07.281627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'cb68b3b3c49f4743bcde95b167282e11', 'request_id': 'req-a8f47591-6137-4ff4-8679-0f31e8e9a66c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:07.289564 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'cb68b3b3c49f4743bcde95b167282e11', 'request_id': 'req-a8f47591-6137-4ff4-8679-0f31e8e9a66c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:07.303073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] The token cache is being invalidate because project cb68b3b3c49f4743bcde95b167282e11 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:07.303073 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:07.303493 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-a8f47591-6137-4ff4-8679-0f31e8e9a66c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:07.305044 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1393/2804] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:07 2022] DELETE /identity/v3/projects/cb68b3b3c49f4743bcde95b167282e11 => generated 0 bytes in 183 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:07.311055 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.311515 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.311610 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.332229 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:07.333469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:07.405010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1412/2805] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 95 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:07.412979 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.413271 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.413552 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.434955 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:07.435153 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:07.443655 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1394/2806] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 32 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:07.448859 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.461558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-44aa79a2-fb73-4d8d-9ba8-75370ca34f0d None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.466747 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.467527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.467722 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.467996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.468757 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:09:07.469062 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.470666 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.482861 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1413/2807] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:09:07 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 35 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:09:07.487741 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.498031 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-0ab78366-97d6-424f-a5af-b248cff502ed None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.502705 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.503397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.503686 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.503965 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.504694 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.506601 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.533627 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1395/2808] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/projects => generated 383 bytes in 46 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:09:07.537900 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.549052 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a497f2dd-a5f5-4ff2-ab43-7c6ceeab2b1f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.607609 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.608285 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.608572 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.608852 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.609547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.611539 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.632128 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1414/2809] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/users => generated 292 bytes in 95 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:07.637496 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.648543 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-31ad7f36-a25d-46ad-ad27-21ef0faa7bab admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.653279 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.653931 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.654219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.654500 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.655268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.656854 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.667702 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1396/2810] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:07.671628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.682326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a40ca107-f532-4693-bad6-a479b7323dc9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.686983 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.687644 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.687927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.688202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.688920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.690520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-e9e74543-1163-40a0-8ec7-1323f46b8cd3 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.700785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1415/2811] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:07 2022] GET /identity/v3/roles => generated 1420 bytes in 30 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:07.704846 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.714706 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ffb2e034-ed34-4441-89d7-5b29bda2b97e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.719348 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.720219 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.720505 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.720783 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] PATH_INFO: `/v3/projects/6ceb22183768439792bfa21b640866e7/users/87b57b3254bf4613a39fbb825c432e86/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.735842 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] RBAC: Authorizing `identity:create_grant(project_id=6ceb22183768439792bfa21b640866e7, user_id=87b57b3254bf4613a39fbb825c432e86, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.738565 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.752657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1397/2812] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:07 2022] PUT /identity/v3/projects/6ceb22183768439792bfa21b640866e7/users/87b57b3254bf4613a39fbb825c432e86/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:07.763547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.763807 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.764244 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.778383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=6,keystone:INSERT=4 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:07.794717 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] MFA Rules not processed for user `87b57b3254bf4613a39fbb825c432e86`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:07.796119 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:07.866029 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1416/2813] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/auth/tokens => generated 3250 bytes in 104 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:07.871170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.933559 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-09bcffdc-ca43-4b5f-ae78-323475534b42 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:07.938270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:07.938954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:07.939240 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:07.939515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:07.940210 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:07.942103 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:07.967830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1398/2814] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/projects => generated 383 bytes in 97 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:07.971947 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:07.983330 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b6f42f64-7fa9-45be-8ec4-e90e76c2abf3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.039944 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.041233 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.041528 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.041803 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.042507 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.044524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.063779 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1417/2815] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:09:07 2022] POST /identity/v3/users => generated 298 bytes in 92 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:08.068078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.078524 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5c1a72c-d656-469a-b15c-86c212eeb2a6 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.083141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.083791 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.084078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.084356 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.085078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.086658 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.088511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=7,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:08.099529 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1399/2816] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:08 2022] GET /identity/v3/roles => generated 1420 bytes in 32 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:08.104374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.114612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ad4c29a8-e09c-4b27-b44b-105cca49823d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.119488 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.120375 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.120657 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.120935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] PATH_INFO: `/v3/projects/1463f56475654eb4960dc3e05b4a4fa3/users/0429d5242ed043d69d5f29669d2bc1a2/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.136812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] RBAC: Authorizing `identity:create_grant(project_id=1463f56475654eb4960dc3e05b4a4fa3, user_id=0429d5242ed043d69d5f29669d2bc1a2, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.139766 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.153817 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1418/2817] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:08 2022] PUT /identity/v3/projects/1463f56475654eb4960dc3e05b4a4fa3/users/0429d5242ed043d69d5f29669d2bc1a2/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 51 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:09:08.157652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.168366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-4f488a61-e152-4af8-bf12-868e083b369b admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.224451 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.225113 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.225397 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.225699 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.226413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.229381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9df2b5c5-3201-40ec-8f7f-71c649ff4165 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.239078 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1400/2818] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:08 2022] GET /identity/v3/roles => generated 1420 bytes in 82 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:08.243394 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.253550 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-58ced20f-3977-4660-abe9-a5f6a47ea507 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.258166 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.259107 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.259392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.259696 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] PATH_INFO: `/v3/projects/1463f56475654eb4960dc3e05b4a4fa3/users/0429d5242ed043d69d5f29669d2bc1a2/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.261629 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] RBAC: Authorizing `identity:create_grant(project_id=1463f56475654eb4960dc3e05b4a4fa3, user_id=0429d5242ed043d69d5f29669d2bc1a2, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.264340 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.274552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1419/2819] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:08 2022] PUT /identity/v3/projects/1463f56475654eb4960dc3e05b4a4fa3/users/0429d5242ed043d69d5f29669d2bc1a2/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 32 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:08.287172 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.287467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.287775 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.315263 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] MFA Rules not processed for user `0429d5242ed043d69d5f29669d2bc1a2`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:08.316378 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:08.384855 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1401/2820] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3317 bytes in 99 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:08.396015 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.397978 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:08.452778 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=95,keystone:INSERT=5 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:08.478470 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-963ce3af-0abb-4d9d-85ae-c1533613697f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.482192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.482863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.483152 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.483428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.492451 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.493884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.503231 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1420/2821] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3317 bytes in 108 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:08.600018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.609726 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5a1769d0-0906-4f59-928e-38ed24612325 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.613315 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.613872 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.614155 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.614434 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.617167 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.618568 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:08.672838 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.674284 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8b2f92eb-7b37-459d-ac4e-4324687a384d service cinder] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.683547 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1402/2822] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3250 bytes in 84 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:08.685383 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-84452841-ae80-4327-ae09-37df0667fa35 service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.689963 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.690865 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.691151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.691424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] PATH_INFO: `/v3/users/4707858b7ef84a7fbd74a93b91cc6e68` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.692924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] RBAC: Authorizing `identity:delete_user(user_id=4707858b7ef84a7fbd74a93b91cc6e68)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.694872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.744440 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '4707858b7ef84a7fbd74a93b91cc6e68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:08.752094 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '4707858b7ef84a7fbd74a93b91cc6e68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:08.752820 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '4707858b7ef84a7fbd74a93b91cc6e68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:08.760654 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '4707858b7ef84a7fbd74a93b91cc6e68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:08.763547 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '4707858b7ef84a7fbd74a93b91cc6e68'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:08.782816 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1421/2823] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:08 2022] DELETE /identity/v3/users/4707858b7ef84a7fbd74a93b91cc6e68 => generated 0 bytes in 166 msecs (HTTP/1.1 204) 3 headers in 132 bytes (1 switches on core 0) Jul 24 01:09:08.787254 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.787546 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.787823 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.804624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:08.806704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:08.816368 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=100 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:08.872767 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1403/2824] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:08 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 87 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:08.882637 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:08.885314 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:08.955283 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a12f2e98-9430-42fc-bcb6-f8c7dc88aeb9 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:08.959287 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:08.960079 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:08.960428 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:08.960767 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:08.967881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:08.972911 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:08.974693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-97ded2ee-e6b2-4752-9a20-5ad9d4eff6ca service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:08.991353 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1422/2825] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:08 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 109 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:09.160554 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.162028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:09.175929 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-faea470b-2fb7-4435-a603-7bfeb89d0284 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.182248 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.183187 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.183592 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.183981 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] PATH_INFO: `/v3/projects/affbcb4611414dcba4e3eca7c0686be7` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.185541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] RBAC: Authorizing `identity:delete_project(project_id=affbcb4611414dcba4e3eca7c0686be7)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.187618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.242830 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'affbcb4611414dcba4e3eca7c0686be7', 'request_id': 'req-3ba3386e-80a6-4b8d-82c7-cfab99c47405'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:09.250004 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'affbcb4611414dcba4e3eca7c0686be7', 'request_id': 'req-3ba3386e-80a6-4b8d-82c7-cfab99c47405'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:09.258096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'affbcb4611414dcba4e3eca7c0686be7', 'request_id': 'req-3ba3386e-80a6-4b8d-82c7-cfab99c47405'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:09.270498 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] The token cache is being invalidate because project affbcb4611414dcba4e3eca7c0686be7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:09.270813 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:09.271454 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3ba3386e-80a6-4b8d-82c7-cfab99c47405 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:09.272997 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1404/2826] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:09 2022] DELETE /identity/v3/projects/affbcb4611414dcba4e3eca7c0686be7 => generated 0 bytes in 113 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:09.278023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.278315 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.278670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.296494 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:09.297694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:09.363050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1423/2827] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 86 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:09.371724 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.372015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.372293 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.390939 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:09.392141 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:09.401003 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1405/2828] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:09:09.405356 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.415797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ea5dd7c3-ea4a-4bd6-a2fb-482f3c2a26b3 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.420579 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.421311 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.421595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.421872 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] PATH_INFO: `/v3/domains` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.422628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80233) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:09:09.422967 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.424594 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.440966 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1424/2829] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:09:09 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 33 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:09:09.442521 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.452444 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8cb05624-b40a-4af3-80cb-a20150c502dd None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.457171 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.457816 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.458096 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.458371 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] PATH_INFO: `/v3/projects` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.459100 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.460984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.485973 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1406/2830] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:09:09 2022] POST /identity/v3/projects => generated 373 bytes in 44 msecs (HTTP/1.1 201) 5 headers in 182 bytes (2 switches on core 0) Jul 24 01:09:09.490110 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.500661 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-86e35265-e1a5-465a-9b9e-09517bf17d6d admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.557487 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.558155 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.558441 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.558715 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] PATH_INFO: `/v3/users` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.559438 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.561357 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.580636 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1425/2831] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:09:09 2022] POST /identity/v3/users => generated 287 bytes in 91 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:09.584675 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.595611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-54b5e38d-26c2-46e2-930f-8c1e4eea2255 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.600260 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.600918 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.601207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.601489 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.602212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.604577 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.615011 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1407/2832] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:09 2022] GET /identity/v3/roles => generated 1420 bytes in 31 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:09.620135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.630390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6aa90c75-4227-4140-8a6d-d04ad825d432 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.635193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.635839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.636118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.636386 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.637126 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.638710 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-2c988a99-541d-41f4-9ed2-12f54a1ebf00 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.648691 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1426/2833] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:09 2022] GET /identity/v3/roles => generated 1420 bytes in 29 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:09.653304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.663573 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-eaa7135c-ee54-4a86-b40d-5a8b5e7d2333 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.668961 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.669869 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.670154 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.670432 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] PATH_INFO: `/v3/projects/6af7b896ae8f4e09957a342a4c0d3030/users/b7a89576949f471f9cd92bbc6677175f/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.685140 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] RBAC: Authorizing `identity:create_grant(project_id=6af7b896ae8f4e09957a342a4c0d3030, user_id=b7a89576949f471f9cd92bbc6677175f, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.687858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.701033 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1408/2834] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:09 2022] PUT /identity/v3/projects/6af7b896ae8f4e09957a342a4c0d3030/users/b7a89576949f471f9cd92bbc6677175f/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 48 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:09.706106 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.706423 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.706698 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.725141 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:09.726347 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:09.776632 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=97,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:09.785679 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1427/2835] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:09 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 81 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:09:09.796119 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:09.797790 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:09.839659 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:09.858685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-16389bac-2c55-48ea-9358-bf1b0c1e573e admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:09.862332 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:09.863018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:09.863308 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:09.863618 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:09.872906 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:09.874292 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5c612a3e-136b-4863-b0f2-136f21de6c67 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:09.883771 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1409/2836] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:09 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 88 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:11.312879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:11.314969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:11.334621 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69a0c556-5c07-4796-8ad8-3e51959add28 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:11.351050 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:11.352146 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:11.352433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:11.352862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] PATH_INFO: `/v3/users/87b57b3254bf4613a39fbb825c432e86` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:11.355348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] RBAC: Authorizing `identity:delete_user(user_id=87b57b3254bf4613a39fbb825c432e86)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:11.357752 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:11.492796 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:11.493390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:11.493390 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:11.535015 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] MFA Rules not processed for user `7c545f7be7794566b639c29c0717c17a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:11.536664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:11.554994 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '87b57b3254bf4613a39fbb825c432e86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:11.575739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '87b57b3254bf4613a39fbb825c432e86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:11.576894 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '87b57b3254bf4613a39fbb825c432e86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:11.590135 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '87b57b3254bf4613a39fbb825c432e86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:11.597310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-1e13c82d-02d7-41b9-a74d-007e4a94a9bb admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '87b57b3254bf4613a39fbb825c432e86'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:11.639154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1428/2837] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:11 2022] DELETE /identity/v3/users/87b57b3254bf4613a39fbb825c432e86 => generated 0 bytes in 326 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:11.644039 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:11.644755 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:11.645208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:11.663664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1410/2838] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:11 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 172 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:11.678142 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:11.680319 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:11.779918 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1429/2839] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:11 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 138 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:09:11.792569 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:11.794484 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:11.809435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-24e72b2e-e980-464b-b50a-f5113fc71793 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:11.894333 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:11.895515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:11.895515 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:11.895928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:11.908018 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:11.909670 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:11.924928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1411/2840] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:11 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 134 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:12.239694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:12.241188 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:12.263390 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b93b9c86-02dc-439c-bb79-ff1b39ab450a None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:12.278426 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:12.280156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:12.280437 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:12.280813 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] PATH_INFO: `/v3/projects/6ceb22183768439792bfa21b640866e7` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:12.282624 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6ceb22183768439792bfa21b640866e7)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:12.284924 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:12.340883 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6ceb22183768439792bfa21b640866e7', 'request_id': 'req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.350023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6ceb22183768439792bfa21b640866e7', 'request_id': 'req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.358270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6ceb22183768439792bfa21b640866e7', 'request_id': 'req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.361567 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=81,keystone:DELETE=17,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:12.375271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] The token cache is being invalidate because project 6ceb22183768439792bfa21b640866e7 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:12.375645 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.376469 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.382366 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1430/2841] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:12 2022] DELETE /identity/v3/projects/6ceb22183768439792bfa21b640866e7 => generated 0 bytes in 141 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:12.383673 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:12.402907 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5bf6beda-8e3d-492b-a929-5d9d328cf761 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:12.479246 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:12.479844 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:12.480134 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:12.480414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] PATH_INFO: `/v3/users/0429d5242ed043d69d5f29669d2bc1a2` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:12.481992 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] RBAC: Authorizing `identity:delete_user(user_id=0429d5242ed043d69d5f29669d2bc1a2)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:12.484055 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:12.519765 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:12.547895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '0429d5242ed043d69d5f29669d2bc1a2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.567511 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '0429d5242ed043d69d5f29669d2bc1a2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.568189 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '0429d5242ed043d69d5f29669d2bc1a2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.577919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '0429d5242ed043d69d5f29669d2bc1a2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.591685 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-6f659dd0-1435-4ac5-8f9a-34ea55b40269 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '0429d5242ed043d69d5f29669d2bc1a2'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:12.598437 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1412/2842] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:12 2022] DELETE /identity/v3/users/0429d5242ed043d69d5f29669d2bc1a2 => generated 0 bytes in 215 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:13.253004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:13.255540 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:13.277628 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9bee09f5-2605-4666-ae1c-1b4a2cab5f52 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:13.376951 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:13.378032 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:13.378320 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:13.378595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] PATH_INFO: `/v3/projects/1463f56475654eb4960dc3e05b4a4fa3` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:13.382563 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] RBAC: Authorizing `identity:delete_project(project_id=1463f56475654eb4960dc3e05b4a4fa3)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:13.387267 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:13.496172 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '1463f56475654eb4960dc3e05b4a4fa3', 'request_id': 'req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:13.507123 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '1463f56475654eb4960dc3e05b4a4fa3', 'request_id': 'req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:13.517380 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '1463f56475654eb4960dc3e05b4a4fa3', 'request_id': 'req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:13.540878 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] The token cache is being invalidate because project 1463f56475654eb4960dc3e05b4a4fa3 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:13.541447 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:13.542228 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-3b1887e0-9e03-4ebf-99bf-cb6885fd7589 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:13.544758 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1431/2843] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:13 2022] DELETE /identity/v3/projects/1463f56475654eb4960dc3e05b4a4fa3 => generated 0 bytes in 292 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:13.549605 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:13.550118 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:13.550220 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:13.583117 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:13.584541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:13.695954 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1413/2844] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 141 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:13.701612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:13.702118 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:13.702553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:13.752467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:13.757017 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:13.787901 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1432/2845] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:13 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 89 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:13.793331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:13.808020 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-98ffd689-c045-4b89-8659-19800f59b43b None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:13.823391 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:13.824093 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:13.824380 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:13.824657 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] PATH_INFO: `/v3/domains` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:13.825422 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] RBAC: Adding query filter params (name=Default) {{(pid=80232) _extract_filter_values /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:167}} Jul 24 01:09:13.825717 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] RBAC: Authorizing `identity:list_domains()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:13.831975 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:13.854644 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1414/2846] 10.222.0.18 () {58 vars in 1152 bytes} [Sun Jul 24 01:09:13 2022] GET /identity/v3/domains?name=Default => generated 308 bytes in 62 msecs (HTTP/1.1 200) 5 headers in 177 bytes (1 switches on core 0) Jul 24 01:09:13.858252 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:13.870881 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f9f08438-b340-4771-a28d-cbeec522ff39 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:13.876719 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:13.877943 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:13.878535 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:13.878919 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:13.879682 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:13.881693 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:13.921467 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1433/2847] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:09:13 2022] POST /identity/v3/projects => generated 377 bytes in 63 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:13.925404 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:13.938285 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-d5e9401e-705a-4569-a228-e39820fa9544 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:14.017600 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:14.018295 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.018580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.018895 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.019812 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:14.021607 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:14.050650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1415/2848] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:09:13 2022] POST /identity/v3/users => generated 289 bytes in 126 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:14.051596 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:14.063272 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-5128db10-aa55-4a97-8239-1f03ef49a06b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:14.069324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:14.069699 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.070100 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.070472 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.071452 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:14.073941 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:14.090970 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1434/2849] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 40 msecs (HTTP/1.1 200) 5 headers in 178 bytes (2 switches on core 0) Jul 24 01:09:14.091989 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:14.104413 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f596838a-aace-448b-8694-b666cae960f0 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:14.110731 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:14.112030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.112030 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.112389 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] PATH_INFO: `/v3/roles` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.113249 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:14.115028 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-69f4206f-6227-44a6-8d17-f0e5a14e3aa3 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:14.127684 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1416/2850] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:14 2022] GET /identity/v3/roles => generated 1420 bytes in 37 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:14.132089 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:14.143321 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-016bc032-8b47-4c7c-aaf4-3029da6a5edf admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:14.152788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:14.154709 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] REQUEST_METHOD: `PUT` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.155255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.155724 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] PATH_INFO: `/v3/projects/a5497514b9d6405eb764b63d809bc939/users/cf986ee700cb4935abe62c142501b257/roles/881cc16b876541819935dfa513a16e5d` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.174808 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] RBAC: Authorizing `identity:create_grant(project_id=a5497514b9d6405eb764b63d809bc939, user_id=cf986ee700cb4935abe62c142501b257, role_id=881cc16b876541819935dfa513a16e5d)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:14.193940 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:14.230842 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1435/2851] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:14 2022] PUT /identity/v3/projects/a5497514b9d6405eb764b63d809bc939/users/cf986ee700cb4935abe62c142501b257/roles/881cc16b876541819935dfa513a16e5d => generated 0 bytes in 89 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:14.234374 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70766792-62c3-458e-9349-2421ca3c8985 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.234833 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70766792-62c3-458e-9349-2421ca3c8985 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.235184 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-70766792-62c3-458e-9349-2421ca3c8985 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.263381 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=91,keystone:DELETE=5,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:14.273758 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-70766792-62c3-458e-9349-2421ca3c8985 None None] MFA Rules not processed for user `cf986ee700cb4935abe62c142501b257`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:14.276313 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-70766792-62c3-458e-9349-2421ca3c8985 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:14.377568 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1417/2852] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:14 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 144 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:14.389588 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:14.391345 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:14.421706 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=90,keystone:DELETE=7,keystone:INSERT=3 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:14.468853 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b33248bd-f160-44f6-a1a4-cdac6f5ebad8 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:14.473746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '5cba3eae172d459bb5c29b712945691c', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:14.474491 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:14.474886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:14.475236 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:14.484908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:14.486530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:14.505760 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1436/2853] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:14 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 117 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:15.273748 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:15.274238 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:15.274361 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:15.316424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] MFA Rules not processed for user `b7a89576949f471f9cd92bbc6677175f`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:15.318364 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:15.425689 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:15.428401 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1418/2854] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:15 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 156 msecs (HTTP/1.1 201) 6 headers in 385 bytes (2 switches on core 0) Jul 24 01:09:15.430131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:15.519324 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-edd9ab08-4d10-40d4-b870-d8f7b87b402f service cinder] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:15.524065 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:15.525595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:15.525946 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:15.526292 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] PATH_INFO: `/v3/projects` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:15.527109 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] RBAC: Authorizing `identity:create_project()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:15.529406 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:15.564255 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1437/2855] 10.222.0.18 () {60 vars in 1152 bytes} [Sun Jul 24 01:09:15 2022] POST /identity/v3/projects => generated 371 bytes in 139 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:15.568894 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:15.582331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c61c4e65-ff21-40f4-97d0-14499bc7ab5d None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:15.684253 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:15.685353 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:15.686182 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:15.686707 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] PATH_INFO: `/v3/users` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:15.687664 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] RBAC: Authorizing `identity:create_user()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:15.693624 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:15.747994 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1419/2856] 10.222.0.18 () {60 vars in 1143 bytes} [Sun Jul 24 01:09:15 2022] POST /identity/v3/users => generated 292 bytes in 180 msecs (HTTP/1.1 201) 5 headers in 182 bytes (1 switches on core 0) Jul 24 01:09:15.752300 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:15.768433 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-472fe091-7b0b-4330-b56e-f8ebc812b72f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:15.773980 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:15.774887 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:15.775419 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:15.775975 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:15.776949 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:15.778746 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:15.792040 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1438/2857] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:15 2022] GET /identity/v3/roles => generated 1420 bytes in 41 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:15.796742 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:15.809467 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-43dab755-370f-4f04-b0fd-c40e78a62495 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:15.819847 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:15.820879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:15.821275 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:15.822424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] PATH_INFO: `/v3/projects/24afdf735d104a73b7f1ad8a92757ffc/users/64c2da5b129d464bb7bd1ae05ae5afc0/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:15.855414 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24afdf735d104a73b7f1ad8a92757ffc, user_id=64c2da5b129d464bb7bd1ae05ae5afc0, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:15.859396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:15.870987 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=96,keystone:INSERT=4 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:15.887385 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1420/2858] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:15 2022] PUT /identity/v3/projects/24afdf735d104a73b7f1ad8a92757ffc/users/64c2da5b129d464bb7bd1ae05ae5afc0/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 91 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:15.892839 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:15.911832 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-21172a74-b889-41a1-a398-51012b2f740b admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:16.028797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:16.029862 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:16.030531 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:16.031025 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] PATH_INFO: `/v3/roles` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:16.031996 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] RBAC: Authorizing `identity:list_roles()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:16.035378 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c9c0b37f-87b8-4c19-8b6a-4f6b5def5672 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:16.055305 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1439/2859] 10.222.0.18 () {58 vars in 1121 bytes} [Sun Jul 24 01:09:15 2022] GET /identity/v3/roles => generated 1420 bytes in 164 msecs (HTTP/1.1 200) 5 headers in 178 bytes (1 switches on core 0) Jul 24 01:09:16.056479 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:16.074237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-855139f5-225e-454e-8d7c-1249804d5fb7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:16.079231 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:16.080302 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] REQUEST_METHOD: `PUT` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:16.080597 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:16.080877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] PATH_INFO: `/v3/projects/24afdf735d104a73b7f1ad8a92757ffc/users/64c2da5b129d464bb7bd1ae05ae5afc0/roles/75e91b1e89304179904cf7644ef0a984` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:16.083174 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] RBAC: Authorizing `identity:create_grant(project_id=24afdf735d104a73b7f1ad8a92757ffc, user_id=64c2da5b129d464bb7bd1ae05ae5afc0, role_id=75e91b1e89304179904cf7644ef0a984)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:16.085856 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:16.096923 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1421/2860] 10.222.0.18 () {60 vars in 1482 bytes} [Sun Jul 24 01:09:16 2022] PUT /identity/v3/projects/24afdf735d104a73b7f1ad8a92757ffc/users/64c2da5b129d464bb7bd1ae05ae5afc0/roles/75e91b1e89304179904cf7644ef0a984 => generated 0 bytes in 42 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:21.399346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17ab6-dae4-4cc4-b9ac-6fdd26f596eb None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:21.399800 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17ab6-dae4-4cc4-b9ac-6fdd26f596eb None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:21.400291 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-21c17ab6-dae4-4cc4-b9ac-6fdd26f596eb None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:21.447318 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-21c17ab6-dae4-4cc4-b9ac-6fdd26f596eb None None] MFA Rules not processed for user `64c2da5b129d464bb7bd1ae05ae5afc0`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:21.454930 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-21c17ab6-dae4-4cc4-b9ac-6fdd26f596eb None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:21.482961 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=99,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:21.534908 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1440/2861] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:21 2022] POST /identity/v3/auth/tokens => generated 3305 bytes in 137 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:21.550991 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:21.553054 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:21.650331 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c705bd56-e032-4193-b88d-e5caecb29167 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:21.656860 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:21.657525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:21.657808 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:21.658084 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:21.718319 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:21.719972 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:21.730048 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1422/2862] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:21 2022] GET /identity/v3/auth/tokens => generated 3240 bytes in 179 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:22.744372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956123a7-1847-4de3-a2bc-e48d48e09f1d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:22.744886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956123a7-1847-4de3-a2bc-e48d48e09f1d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:22.744886 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-956123a7-1847-4de3-a2bc-e48d48e09f1d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:22.762675 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-956123a7-1847-4de3-a2bc-e48d48e09f1d None None] MFA Rules not processed for user `b7a89576949f471f9cd92bbc6677175f`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:22.763927 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-956123a7-1847-4de3-a2bc-e48d48e09f1d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:22.772421 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1441/2863] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:22 2022] POST /identity/v3/auth/tokens => generated 3240 bytes in 30 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:31.725035 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=63,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:32.757154 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=21 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:34.827469 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:34.892107 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c95378e-dc41-4957-8c4e-f714970a3696 service nova] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:34.896998 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:34.898480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:34.898480 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:34.898716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] PATH_INFO: `/v3/users/cf986ee700cb4935abe62c142501b257` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:34.900017 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] RBAC: Authorizing `identity:delete_user(user_id=cf986ee700cb4935abe62c142501b257)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:34.902159 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:34.955897 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'cf986ee700cb4935abe62c142501b257'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:34.972116 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'cf986ee700cb4935abe62c142501b257'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:34.972788 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'cf986ee700cb4935abe62c142501b257'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:34.982287 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'cf986ee700cb4935abe62c142501b257'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:34.996268 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'cf986ee700cb4935abe62c142501b257'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.008676 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1423/2864] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:09:34 2022] DELETE /identity/v3/users/cf986ee700cb4935abe62c142501b257 => generated 0 bytes in 182 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:35.013559 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:35.014042 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:35.014165 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:35.031618 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] MFA Rules not processed for user `f00ceffcf1cf4b95908b3794e7b9b31d`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:09:35.032797 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:35.097476 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1442/2865] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:09:35 2022] POST /identity/v3/auth/tokens => generated 3229 bytes in 85 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:09:35.108421 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:35.110039 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:35.171494 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-20203d04-acbf-460c-851a-8719daa65f4c admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:35.175456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:35.176191 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:35.176513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:35.176968 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:35.186747 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:35.188213 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:35.198112 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1424/2866] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:35 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 91 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:35.525585 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:35.527530 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:35.539822 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3c584d76-1511-480a-8dc7-404b46e26dfe None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:35.545592 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:35.546346 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:35.546577 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:35.546879 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] PATH_INFO: `/v3/projects/a5497514b9d6405eb764b63d809bc939` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:35.548075 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] RBAC: Authorizing `identity:delete_project(project_id=a5497514b9d6405eb764b63d809bc939)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:35.550004 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:35.606234 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'a5497514b9d6405eb764b63d809bc939', 'request_id': 'req-c037682e-a7e9-47b7-a240-4a35bec54cfd'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.613700 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'a5497514b9d6405eb764b63d809bc939', 'request_id': 'req-c037682e-a7e9-47b7-a240-4a35bec54cfd'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.619614 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'a5497514b9d6405eb764b63d809bc939', 'request_id': 'req-c037682e-a7e9-47b7-a240-4a35bec54cfd'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.630524 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] The token cache is being invalidate because project a5497514b9d6405eb764b63d809bc939 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80233) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:09:35.630845 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.631481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:09:35.633034 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1443/2867] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:09:35 2022] DELETE /identity/v3/projects/a5497514b9d6405eb764b63d809bc939 => generated 0 bytes in 109 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:09:36.951126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:36.952571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:36.964354 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-53583dd0-73e6-464e-8fbd-64e838d53a65 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:36.982885 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:37.015588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:37.021501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:37.021501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:37.021501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:37.021501 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:37.083497 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:37.084977 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:37.094598 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1425/2868] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:36 2022] GET /identity/v3/auth/tokens => generated 3313 bytes in 144 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:09:45.627553 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=54,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:47.090326 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=40 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:09:54.942880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:09:54.944270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:54.956663 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-c037682e-a7e9-47b7-a240-4a35bec54cfd admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:09:54.960193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:09:54.960826 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:09:54.961103 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:09:54.961377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:09:54.963358 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:09:55.025608 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:09:55.027060 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:09:55.036387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1444/2869] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:09:54 2022] GET /identity/v3/auth/tokens => generated 3305 bytes in 94 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:10:05.032202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=30 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:11:36.099571 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:36.108270 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b1d707e9-fd43-4de4-ba66-12df56b280b0 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:36.112094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:36.112715 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:36.113000 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:36.113276 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:36.115049 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:36.185098 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:36.186456 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:36.198298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1426/2870] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:36 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 99 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:37.376932 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:37.385640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f3a4bbf7-6f01-4cd5-9e91-72bc58db3996 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:37.389281 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:37.389914 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:37.390192 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:37.391169 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:37.393206 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:37.407264 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:37.409202 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-766c7c63-d585-440f-9170-7d1237a04021 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:37.420495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1445/2871] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:37 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 44 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:38.540580 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:38.542237 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:38.571722 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fd45e5e4-7d27-411d-92d3-2d4056a523a3 service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:38.584415 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:38.585202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:38.585492 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:38.585773 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:38.588473 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:38.609321 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:38.611063 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:38.624527 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1427/2872] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:38 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 85 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:39.692342 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:39.693179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:39.693179 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:39.711364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] MFA Rules not processed for user `7c545f7be7794566b639c29c0717c17a`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:11:39.712587 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:39.768676 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1446/2873] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:11:39 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 77 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:11:40.514481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:40.515984 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:40.586130 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-41b395c4-2a79-47a0-bbbc-bfbd49d736b6 service nova] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:40.591094 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:40.591739 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:40.592025 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:40.592306 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:40.594621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:40.605792 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:40.607439 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:40.618058 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1428/2874] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:40 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 104 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:42.307581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:42.310023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:42.324478 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3b0bc668-4978-4958-9030-9dd4ba05d5a9 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:42.329145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:42.330010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:42.330510 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:42.331016 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:42.332882 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:42.344519 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:42.346957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:42.356702 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1447/2875] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:42 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:50.613661 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=72 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:11:52.352377 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=37 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:11:53.563207 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:53.577419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56868ba5-db47-4629-a994-4326661f9bbb service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:53.583639 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:53.584289 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:53.584579 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:53.584877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:53.586631 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:53.599781 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:53.601366 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-81744518-1811-49e0-95c7-cf8cc723e9c2 service nova] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:53.613621 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1429/2876] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:53 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 51 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:11:57.796344 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:11:57.799481 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:57.818741 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6f55915f-a2e9-4f60-8648-1ba44b1e5921 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:11:57.824280 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:11:57.824923 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:11:57.825208 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:11:57.825483 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:11:57.827310 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:11:57.840168 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:11:57.841761 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-704beaba-b2f1-491e-8092-293ebaba3895 service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:11:57.853145 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1448/2877] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:11:57 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 57 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:12:03.608394 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=6 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:12:07.306540 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47338a1-58a5-49b1-a2fd-a368569f7a36 None None] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:07.307281 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47338a1-58a5-49b1-a2fd-a368569f7a36 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:07.307447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-b47338a1-58a5-49b1-a2fd-a368569f7a36 None None] PATH_INFO: `/` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:07.309165 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1430/2878] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:12:07 2022] GET /identity => generated 270 bytes in 4 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:12:07.316694 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabb643-9a58-4fbf-b134-6e08c2f7ad3d None None] REQUEST_METHOD: `POST` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:07.317270 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabb643-9a58-4fbf-b134-6e08c2f7ad3d None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:07.317387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6eabb643-9a58-4fbf-b134-6e08c2f7ad3d None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:07.354595 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.auth.core [None req-6eabb643-9a58-4fbf-b134-6e08c2f7ad3d None None] MFA Rules not processed for user `e8144687e0b546108a7a5eb55ac9b452`. Rule list: `[]` (Enabled: `True`). {{(pid=80233) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:12:07.355858 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-6eabb643-9a58-4fbf-b134-6e08c2f7ad3d None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:12:07.362810 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1449/2879] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:12:07 2022] POST /identity/v3/auth/tokens => generated 3111 bytes in 47 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:12:07.375526 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47338a1-58a5-49b1-a2fd-a368569f7a36 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:12:07.387704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-b47338a1-58a5-49b1-a2fd-a368569f7a36 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:12:07.391449 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:12:07.392768 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:07.393089 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:07.393841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:07.404976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:12:07.406424 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-206ae8e5-f6be-4434-b779-4e3231cfa37c service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:12:07.416336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1431/2880] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:12:07 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 41 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:12:08.506972 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d174534-36df-45c8-8270-09734ad6a545 None None] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:08.507454 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d174534-36df-45c8-8270-09734ad6a545 None None] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:08.507573 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-6d174534-36df-45c8-8270-09734ad6a545 None None] PATH_INFO: `/` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:08.508880 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1450/2881] 10.222.0.18 () {54 vars in 921 bytes} [Sun Jul 24 01:12:08 2022] GET /identity => generated 270 bytes in 3 msecs (HTTP/1.1 300) 6 headers in 234 bytes (1 switches on core 0) Jul 24 01:12:08.514704 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc4608-987d-4b61-a134-d7851fc41dcc None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:08.515185 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc4608-987d-4b61-a134-d7851fc41dcc None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:08.515303 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-50fc4608-987d-4b61-a134-d7851fc41dcc None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:08.551953 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-50fc4608-987d-4b61-a134-d7851fc41dcc None None] MFA Rules not processed for user `865a61591b7d45ca90431a7683e96a61`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:12:08.553177 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-50fc4608-987d-4b61-a134-d7851fc41dcc None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:12:08.562506 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1432/2882] 10.222.0.18 () {58 vars in 1018 bytes} [Sun Jul 24 01:12:08 2022] POST /identity/v3/auth/tokens => generated 3362 bytes in 49 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:12:08.574271 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d174534-36df-45c8-8270-09734ad6a545 None None] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:12:08.585863 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-6d174534-36df-45c8-8270-09734ad6a545 None None] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:12:08.589395 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:12:08.590027 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:12:08.590307 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:12:08.590581 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:12:08.601732 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:12:08.603195 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:12:08.615293 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1451/2883] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:12:08 2022] GET /identity/v3/auth/tokens => generated 3362 bytes in 42 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:12:18.549513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=15 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:12:18.609389 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=21 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:13:43.586610 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] REQUEST_METHOD: `POST` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:43.587227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:43.587227 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:43.605314 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.auth.core [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] MFA Rules not processed for user `7c545f7be7794566b639c29c0717c17a`. Rule list: `[]` (Enabled: `True`). {{(pid=80232) check_auth_methods_against_rules /opt/stack/keystone/keystone/auth/core.py:438}} Jul 24 01:13:43.606513 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:43.613541 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1433/2884] 10.222.0.18 () {58 vars in 957 bytes} [Sun Jul 24 01:13:43 2022] POST /identity/v3/auth/tokens => generated 3244 bytes in 28 msecs (HTTP/1.1 201) 6 headers in 385 bytes (1 switches on core 0) Jul 24 01:13:44.004259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:44.012873 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-95872f90-8519-4d94-919f-3c4947d5c756 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:44.016463 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:44.017090 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:44.017370 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:44.017642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:44.020081 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:44.029466 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:44.030957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:44.039343 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1452/2885] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:13:44 2022] GET /identity/v3/auth/tokens => generated 3111 bytes in 36 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:13:49.702950 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:49.704340 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:49.732852 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-9e1cb0e3-4c7f-49bb-b9ad-83f3d5d37c45 None None] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:49.739221 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:49.740170 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:49.740525 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:49.740859 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:49.743396 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:49.756703 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:49.758304 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:49.779244 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1434/2886] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:13:49 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 77 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:13:51.654495 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:51.669374 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-11aa9599-79b9-4d1d-b86e-8e3e29d0d2b7 service neutron] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:51.675387 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '865a61591b7d45ca90431a7683e96a61', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'service', 'admin', 'ResellerAdmin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:51.676248 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:51.676739 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:51.677023 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:51.678851 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:51.692348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:51.693950 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:51.704045 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1453/2887] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:13:51 2022] GET /identity/v3/auth/tokens => generated 3244 bytes in 50 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:13:52.813787 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:52.826377 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-59cc9149-32b6-4039-9166-e4294e3e7ea1 service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:52.832588 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:52.833252 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:52.833533 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:52.833809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:52.835824 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:52.849355 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:52.851620 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:52.865641 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1435/2888] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:13:52 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 52 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:13:52.887620 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:52.889156 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:52.961734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f5483d9e-e88b-4168-93f9-ed7c189d979c service nova] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:52.966313 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:52.967216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:52.967520 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:52.967794 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] PATH_INFO: `/v3/users/b7a89576949f471f9cd92bbc6677175f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:52.968965 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] RBAC: Authorizing `identity:delete_user(user_id=b7a89576949f471f9cd92bbc6677175f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:52.970929 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:53.019670 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'b7a89576949f471f9cd92bbc6677175f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.027759 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'b7a89576949f471f9cd92bbc6677175f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.028527 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'b7a89576949f471f9cd92bbc6677175f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.036398 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'b7a89576949f471f9cd92bbc6677175f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.037053 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'b7a89576949f471f9cd92bbc6677175f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.055424 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1454/2889] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:13:52 2022] DELETE /identity/v3/users/b7a89576949f471f9cd92bbc6677175f => generated 0 bytes in 168 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:13:53.227802 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:53.238867 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-56f0beff-f3d8-4e3f-a6f0-b461de0495b8 service placement] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:53.298879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:53.299749 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:53.300037 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:53.300919 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] PATH_INFO: `/v3/projects/6af7b896ae8f4e09957a342a4c0d3030` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:53.302247 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] RBAC: Authorizing `identity:delete_project(project_id=6af7b896ae8f4e09957a342a4c0d3030)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:53.304202 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:53.349877 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '6af7b896ae8f4e09957a342a4c0d3030', 'request_id': 'req-3356db03-279d-4cfd-a376-8e10c31e0a35'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.355214 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '6af7b896ae8f4e09957a342a4c0d3030', 'request_id': 'req-3356db03-279d-4cfd-a376-8e10c31e0a35'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.362150 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '6af7b896ae8f4e09957a342a4c0d3030', 'request_id': 'req-3356db03-279d-4cfd-a376-8e10c31e0a35'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.373423 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] The token cache is being invalidate because project 6af7b896ae8f4e09957a342a4c0d3030 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:13:53.373716 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.374349 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.375928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1436/2890] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:13:53 2022] DELETE /identity/v3/projects/6af7b896ae8f4e09957a342a4c0d3030 => generated 0 bytes in 149 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:13:53.380111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:53.381551 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:53.395014 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-a2d4648c-f41f-4589-8ea4-fdf7353f8a7f admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:53.442844 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:13:53.453294 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:53.454177 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:53.454462 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:53.454738 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] PATH_INFO: `/v3/users/64c2da5b129d464bb7bd1ae05ae5afc0` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:53.455981 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] RBAC: Authorizing `identity:delete_user(user_id=64c2da5b129d464bb7bd1ae05ae5afc0)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:53.457957 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:53.505435 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '64c2da5b129d464bb7bd1ae05ae5afc0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.511998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '64c2da5b129d464bb7bd1ae05ae5afc0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.512612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '64c2da5b129d464bb7bd1ae05ae5afc0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.519537 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '64c2da5b129d464bb7bd1ae05ae5afc0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.520151 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '64c2da5b129d464bb7bd1ae05ae5afc0'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.536989 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1455/2891] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:13:53 2022] DELETE /identity/v3/users/64c2da5b129d464bb7bd1ae05ae5afc0 => generated 0 bytes in 157 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:13:53.717669 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:13:53.719457 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:13:53.735230 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-3356db03-279d-4cfd-a376-8e10c31e0a35 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:13:53.787336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=92,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:13:53.794754 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:13:53.795841 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:13:53.796307 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:13:53.796650 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] PATH_INFO: `/v3/projects/24afdf735d104a73b7f1ad8a92757ffc` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:13:53.797917 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] RBAC: Authorizing `identity:delete_project(project_id=24afdf735d104a73b7f1ad8a92757ffc)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:13:53.800201 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:13:53.845766 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': '24afdf735d104a73b7f1ad8a92757ffc', 'request_id': 'req-dedeba7b-3516-42bb-9c94-d745d8ee71d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.851806 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': '24afdf735d104a73b7f1ad8a92757ffc', 'request_id': 'req-dedeba7b-3516-42bb-9c94-d745d8ee71d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.858962 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': '24afdf735d104a73b7f1ad8a92757ffc', 'request_id': 'req-dedeba7b-3516-42bb-9c94-d745d8ee71d7'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.870417 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] The token cache is being invalidate because project 24afdf735d104a73b7f1ad8a92757ffc was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:13:53.870719 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.871359 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:13:53.872880 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1437/2892] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:13:53 2022] DELETE /identity/v3/projects/24afdf735d104a73b7f1ad8a92757ffc => generated 0 bytes in 156 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:14:03.533348 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=29,keystone:DELETE=10,keystone:INSERT=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:14:03.867481 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=25,keystone:DELETE=7,keystone:INSERT=1 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:14:48.619617 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:14:48.621049 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:14:48.690489 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8f010996-083f-42a4-99e3-2622a62b0806 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:14:48.748785 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:14:48.749431 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:14:48.749714 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:14:48.749987 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:14:48.751788 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:14:48.762734 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:14:48.764352 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:14:48.774187 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1456/2893] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:14:48 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 155 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:14:58.770010 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=58 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:15:42.974126 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:15:42.975652 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:15:43.032419 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-dedeba7b-3516-42bb-9c94-d745d8ee71d7 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:15:43.035970 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'e8144687e0b546108a7a5eb55ac9b452', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['service'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:15:43.036612 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] REQUEST_METHOD: `GET` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:15:43.036898 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:15:43.037176 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] PATH_INFO: `/v3/auth/tokens` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:15:43.039611 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:15:43.101338 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] RBAC: Authorizing `identity:validate_token()` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:15:43.103309 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:15:43.112696 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1438/2894] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:15:42 2022] GET /identity/v3/auth/tokens => generated 3229 bytes in 139 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:15:49.337295 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:15:49.338935 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:15:49.365224 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-05824332-1aed-4dc7-9634-03ed37585fd5 service placement] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:15:49.370409 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:15:49.371396 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:15:49.371705 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:15:49.371971 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] PATH_INFO: `/v3/users/7c545f7be7794566b639c29c0717c17a` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:15:49.373159 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] RBAC: Authorizing `identity:delete_user(user_id=7c545f7be7794566b639c29c0717c17a)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:15:49.375131 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:15:49.422364 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': '7c545f7be7794566b639c29c0717c17a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.429612 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': '7c545f7be7794566b639c29c0717c17a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.430420 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': '7c545f7be7794566b639c29c0717c17a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.437259 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': '7c545f7be7794566b639c29c0717c17a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.437884 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': '7c545f7be7794566b639c29c0717c17a'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.455351 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1457/2895] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:15:49 2022] DELETE /identity/v3/users/7c545f7be7794566b639c29c0717c17a => generated 0 bytes in 119 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:15:49.635535 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:15:49.645809 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-8908cf18-ac34-4f9c-ab7a-959bcd3b2b9d service neutron] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:15:49.710478 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:15:49.711922 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:15:49.712212 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:15:49.712491 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] PATH_INFO: `/v3/projects/be64d6ea0f0149608c9a823425a0e811` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:15:49.713928 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] RBAC: Authorizing `identity:delete_project(project_id=be64d6ea0f0149608c9a823425a0e811)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:15:49.715858 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:15:49.767400 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'be64d6ea0f0149608c9a823425a0e811', 'request_id': 'req-ba386fa1-66ff-478e-b63a-b53fa9675cf5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.769296 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=95,keystone:DELETE=5 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:15:49.774447 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'be64d6ea0f0149608c9a823425a0e811', 'request_id': 'req-ba386fa1-66ff-478e-b63a-b53fa9675cf5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.782572 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'be64d6ea0f0149608c9a823425a0e811', 'request_id': 'req-ba386fa1-66ff-478e-b63a-b53fa9675cf5'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.794393 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] The token cache is being invalidate because project be64d6ea0f0149608c9a823425a0e811 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:15:49.794678 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.795336 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.796879 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1439/2896] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:15:49 2022] DELETE /identity/v3/projects/be64d6ea0f0149608c9a823425a0e811 => generated 0 bytes in 162 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:15:49.800920 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:15:49.802326 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:15:49.815508 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-421c14ed-6515-41b6-9c45-0b42449d2337 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:15:49.872642 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:15:49.873448 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] REQUEST_METHOD: `DELETE` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:15:49.873725 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:15:49.873998 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] PATH_INFO: `/v3/users/a8c1e286d5ad40dc89ffb9baca32925f` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:15:49.875203 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] RBAC: Authorizing `identity:delete_user(user_id=a8c1e286d5ad40dc89ffb9baca32925f)` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:15:49.877111 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:15:49.922818 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Invoking callback _user_callback for event identity user deleted for {'resource_info': 'a8c1e286d5ad40dc89ffb9baca32925f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.930392 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Invoking callback _drop_token_cache for event identity user deleted for {'resource_info': 'a8c1e286d5ad40dc89ffb9baca32925f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.931099 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Invoking callback _on_user_delete for event identity user deleted for {'resource_info': 'a8c1e286d5ad40dc89ffb9baca32925f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.937979 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Invoking callback _drop_receipt_cache for event identity user deleted for {'resource_info': 'a8c1e286d5ad40dc89ffb9baca32925f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.938625 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.notifications [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Invoking callback _delete_app_creds_on_user_delete_callback for event identity user deleted for {'resource_info': 'a8c1e286d5ad40dc89ffb9baca32925f'} {{(pid=80233) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:49.947193 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=79,keystone:DELETE=19,keystone:INSERT=2 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:15:49.958216 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1458/2897] 10.222.0.18 () {58 vars in 1223 bytes} [Sun Jul 24 01:15:49 2022] DELETE /identity/v3/users/a8c1e286d5ad40dc89ffb9baca32925f => generated 0 bytes in 158 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:15:50.125988 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Authenticating user token {{(pid=80232) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:15:50.127435 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.fernet_utils [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80232) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:15:50.141729 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-ba386fa1-66ff-478e-b63a-b53fa9675cf5 admin admin] Validating token access rules against request {{(pid=80232) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:15:50.325591 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': 'f00ceffcf1cf4b95908b3794e7b9b31d', 'user_domain_id': 'default', 'system_scope': None, 'project_id': '17ae0b4181524f419070ad7610eaf309', 'project_domain_id': 'default', 'roles': ['reader', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80232) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:15:50.326512 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] REQUEST_METHOD: `DELETE` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:15:50.326882 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] SCRIPT_NAME: `/identity` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:15:50.328091 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.server.flask.request_processing.req_logging [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] PATH_INFO: `/v3/projects/c41a498067354c0a81f56e90db1da183` {{(pid=80232) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:15:50.328976 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] RBAC: Authorizing `identity:delete_project(project_id=c41a498067354c0a81f56e90db1da183)` {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:15:50.330956 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] RBAC: Authorization granted {{(pid=80232) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:15:50.376671 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] Invoking callback _on_project_or_endpoint_delete for event identity project deleted for {'resource_info': 'c41a498067354c0a81f56e90db1da183', 'request_id': 'req-cd93f363-1821-483b-a6b1-800a2103063c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:50.382286 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] Invoking callback _project_callback for event identity project deleted for {'resource_info': 'c41a498067354c0a81f56e90db1da183', 'request_id': 'req-cd93f363-1821-483b-a6b1-800a2103063c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:50.390357 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] Invoking callback _unset_default_project for event identity project deleted for {'resource_info': 'c41a498067354c0a81f56e90db1da183', 'request_id': 'req-cd93f363-1821-483b-a6b1-800a2103063c'} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:50.401762 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] The token cache is being invalidate because project c41a498067354c0a81f56e90db1da183 was deleted. Authorization will be recalculated and enforced accordingly the next time users authenticate or validate a token. {{(pid=80232) invalidate_token_cache_notification /opt/stack/keystone/keystone/notifications.py:230}} Jul 24 01:15:50.402052 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] Invoking callback _drop_token_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:50.402700 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG keystone.notifications [None req-cd93f363-1821-483b-a6b1-800a2103063c admin admin] Invoking callback _drop_receipt_cache for event identity invalidate_token_cache internal for {'resource_info': None} {{(pid=80232) notify_event_callbacks /opt/stack/keystone/keystone/notifications.py:355}} Jul 24 01:15:50.404298 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: [pid: 80232|app: 0|req: 1440/2898] 10.222.0.18 () {58 vars in 1232 bytes} [Sun Jul 24 01:15:50 2022] DELETE /identity/v3/projects/c41a498067354c0a81f56e90db1da183 => generated 0 bytes in 279 msecs (HTTP/1.1 204) 3 headers in 132 bytes (0 switches on core 0) Jul 24 01:15:59.956443 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=3,keystone:DELETE=1 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:16:00.398445 dv-cinder-master-3-iscsi devstack@keystone.service[80232]: DEBUG dbcounter [-] [80232] Writing DB stats keystone:SELECT=57,keystone:DELETE=9,keystone:INSERT=2 {{(pid=80232) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}} Jul 24 01:19:51.555552 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Authenticating user token {{(pid=80233) process_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:407}} Jul 24 01:19:51.556968 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:19:51.699615 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-fbb8c6e1-7c55-45ae-8d3b-52d440e3c973 admin admin] Validating token access rules against request {{(pid=80233) validate_allowed_request /usr/local/lib/python3.8/dist-packages/keystonemiddleware/auth_token/__init__.py:545}} Jul 24 01:19:51.757558 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.middleware.auth_context [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] RBAC: auth_context: {'token': , 'domain_id': None, 'trust_id': None, 'trustor_id': None, 'trustee_id': None, 'domain_name': None, 'group_ids': [], 'user_id': '9eaff003925c4f7283d99ad51db217e2', 'user_domain_id': 'default', 'system_scope': None, 'project_id': 'ca6929ff61c44a8bb05ab68c6be32354', 'project_domain_id': 'default', 'roles': ['reader', 'service', 'member', 'admin'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} {{(pid=80233) fill_context /opt/stack/keystone/keystone/server/flask/request_processing/middleware/auth_context.py:478}} Jul 24 01:19:51.758210 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] REQUEST_METHOD: `GET` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:27}} Jul 24 01:19:51.758499 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] SCRIPT_NAME: `/identity` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:28}} Jul 24 01:19:51.758812 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.server.flask.request_processing.req_logging [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] PATH_INFO: `/v3/auth/tokens` {{(pid=80233) log_request_info /opt/stack/keystone/keystone/server/flask/request_processing/req_logging.py:29}} Jul 24 01:19:51.760590 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.fernet_utils [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] Loaded 2 Fernet keys from /etc/keystone/fernet-keys/, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? {{(pid=80233) load_keys /opt/stack/keystone/keystone/common/fernet_utils.py:286}} Jul 24 01:19:51.772372 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] RBAC: Authorizing `identity:validate_token()` {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:449}} Jul 24 01:19:51.773969 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG keystone.common.rbac_enforcer.enforcer [None req-f534fabb-2913-4ad0-a472-7fba0368b451 service placement] RBAC: Authorization granted {{(pid=80233) enforce_call /opt/stack/keystone/keystone/common/rbac_enforcer/enforcer.py:457}} Jul 24 01:19:51.784578 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: [pid: 80233|app: 0|req: 1459/2899] 10.222.0.18 () {60 vars in 1365 bytes} [Sun Jul 24 01:19:51 2022] GET /identity/v3/auth/tokens => generated 3298 bytes in 230 msecs (HTTP/1.1 200) 6 headers in 380 bytes (1 switches on core 0) Jul 24 01:20:01.779640 dv-cinder-master-3-iscsi devstack@keystone.service[80233]: DEBUG dbcounter [-] [80233] Writing DB stats keystone:SELECT=89 {{(pid=80233) stat_writer /usr/local/lib/python3.8/dist-packages/dbcounter.py:114}}